2023-04-05 04:56:04,604 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bbb152a5-abc2-1c00-5587-f0792a401dc1 2023-04-05 04:56:04,615 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-05 04:56:04,629 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-05 04:56:04,630 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bbb152a5-abc2-1c00-5587-f0792a401dc1/cluster_90a8a3c0-d608-d540-0120-10832eef50d1, deleteOnExit=true 2023-04-05 04:56:04,630 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-05 04:56:04,631 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bbb152a5-abc2-1c00-5587-f0792a401dc1/test.cache.data in system properties and HBase conf 2023-04-05 04:56:04,631 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bbb152a5-abc2-1c00-5587-f0792a401dc1/hadoop.tmp.dir in system properties and HBase conf 2023-04-05 04:56:04,631 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bbb152a5-abc2-1c00-5587-f0792a401dc1/hadoop.log.dir in system properties and HBase conf 2023-04-05 04:56:04,632 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bbb152a5-abc2-1c00-5587-f0792a401dc1/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-05 04:56:04,633 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bbb152a5-abc2-1c00-5587-f0792a401dc1/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-05 04:56:04,633 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-05 04:56:04,723 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-05 04:56:05,069 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-05 04:56:05,072 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bbb152a5-abc2-1c00-5587-f0792a401dc1/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-05 04:56:05,072 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bbb152a5-abc2-1c00-5587-f0792a401dc1/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-05 04:56:05,073 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bbb152a5-abc2-1c00-5587-f0792a401dc1/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-05 04:56:05,073 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bbb152a5-abc2-1c00-5587-f0792a401dc1/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-05 04:56:05,073 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bbb152a5-abc2-1c00-5587-f0792a401dc1/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-05 04:56:05,074 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bbb152a5-abc2-1c00-5587-f0792a401dc1/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-05 04:56:05,074 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bbb152a5-abc2-1c00-5587-f0792a401dc1/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-05 04:56:05,074 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bbb152a5-abc2-1c00-5587-f0792a401dc1/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-05 04:56:05,074 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bbb152a5-abc2-1c00-5587-f0792a401dc1/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-05 04:56:05,075 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bbb152a5-abc2-1c00-5587-f0792a401dc1/nfs.dump.dir in system properties and HBase conf 2023-04-05 04:56:05,075 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bbb152a5-abc2-1c00-5587-f0792a401dc1/java.io.tmpdir in system properties and HBase conf 2023-04-05 04:56:05,075 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bbb152a5-abc2-1c00-5587-f0792a401dc1/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-05 04:56:05,075 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bbb152a5-abc2-1c00-5587-f0792a401dc1/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-05 04:56:05,076 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bbb152a5-abc2-1c00-5587-f0792a401dc1/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-05 04:56:05,470 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-05 04:56:05,474 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-05 04:56:05,693 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-05 04:56:05,824 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-05 04:56:05,838 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-05 04:56:05,869 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-05 04:56:05,895 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bbb152a5-abc2-1c00-5587-f0792a401dc1/java.io.tmpdir/Jetty_localhost_localdomain_35045_hdfs____.tqvifr/webapp 2023-04-05 04:56:05,998 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:35045 2023-04-05 04:56:06,030 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-05 04:56:06,031 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-05 04:56:06,334 WARN [Listener at localhost.localdomain/39291] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-05 04:56:06,418 WARN [Listener at localhost.localdomain/39291] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-05 04:56:06,434 WARN [Listener at localhost.localdomain/39291] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-05 04:56:06,439 INFO [Listener at localhost.localdomain/39291] log.Slf4jLog(67): jetty-6.1.26 2023-04-05 04:56:06,443 INFO [Listener at localhost.localdomain/39291] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bbb152a5-abc2-1c00-5587-f0792a401dc1/java.io.tmpdir/Jetty_localhost_33581_datanode____icjyid/webapp 2023-04-05 04:56:06,518 INFO [Listener at localhost.localdomain/39291] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33581 2023-04-05 04:56:06,786 WARN [Listener at localhost.localdomain/34579] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-05 04:56:07,150 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x285a218593b19f11: Processing first storage report for DS-67667659-cf10-4833-b2f2-a62752fe1d31 from datanode 6c6dadc0-bae8-4933-9020-feb125a9e7fa 2023-04-05 04:56:07,151 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x285a218593b19f11: from storage DS-67667659-cf10-4833-b2f2-a62752fe1d31 node DatanodeRegistration(127.0.0.1:45635, datanodeUuid=6c6dadc0-bae8-4933-9020-feb125a9e7fa, infoPort=35789, infoSecurePort=0, ipcPort=34579, storageInfo=lv=-57;cid=testClusterID;nsid=217902131;c=1680670565529), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-04-05 04:56:07,151 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x285a218593b19f11: Processing first storage report for DS-4922f251-1a4a-4816-8854-6d6cff13d8a5 from datanode 6c6dadc0-bae8-4933-9020-feb125a9e7fa 2023-04-05 04:56:07,152 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x285a218593b19f11: from storage DS-4922f251-1a4a-4816-8854-6d6cff13d8a5 node DatanodeRegistration(127.0.0.1:45635, datanodeUuid=6c6dadc0-bae8-4933-9020-feb125a9e7fa, infoPort=35789, infoSecurePort=0, ipcPort=34579, storageInfo=lv=-57;cid=testClusterID;nsid=217902131;c=1680670565529), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-05 04:56:07,335 DEBUG [Listener at localhost.localdomain/34579] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bbb152a5-abc2-1c00-5587-f0792a401dc1 2023-04-05 04:56:07,394 INFO [Listener at localhost.localdomain/34579] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bbb152a5-abc2-1c00-5587-f0792a401dc1/cluster_90a8a3c0-d608-d540-0120-10832eef50d1/zookeeper_0, clientPort=56229, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bbb152a5-abc2-1c00-5587-f0792a401dc1/cluster_90a8a3c0-d608-d540-0120-10832eef50d1/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bbb152a5-abc2-1c00-5587-f0792a401dc1/cluster_90a8a3c0-d608-d540-0120-10832eef50d1/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-05 04:56:07,404 INFO [Listener at localhost.localdomain/34579] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=56229 2023-04-05 04:56:07,413 INFO [Listener at localhost.localdomain/34579] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-05 04:56:07,415 INFO [Listener at localhost.localdomain/34579] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-05 04:56:08,017 INFO [Listener at localhost.localdomain/34579] util.FSUtils(471): Created version file at hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7 with version=8 2023-04-05 04:56:08,017 INFO [Listener at localhost.localdomain/34579] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/hbase-staging 2023-04-05 04:56:08,290 INFO [Listener at localhost.localdomain/34579] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-05 04:56:08,694 INFO [Listener at localhost.localdomain/34579] client.ConnectionUtils(127): master/jenkins-hbase20:0 server-side Connection retries=45 2023-04-05 04:56:08,722 INFO [Listener at localhost.localdomain/34579] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-05 04:56:08,723 INFO [Listener at localhost.localdomain/34579] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-05 04:56:08,723 INFO [Listener at localhost.localdomain/34579] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-05 04:56:08,723 INFO [Listener at localhost.localdomain/34579] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-05 04:56:08,723 INFO [Listener at localhost.localdomain/34579] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-05 04:56:08,868 INFO [Listener at localhost.localdomain/34579] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-05 04:56:08,939 DEBUG [Listener at localhost.localdomain/34579] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-05 04:56:09,027 INFO [Listener at localhost.localdomain/34579] ipc.NettyRpcServer(120): Bind to /148.251.75.209:43797 2023-04-05 04:56:09,038 INFO [Listener at localhost.localdomain/34579] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-05 04:56:09,039 INFO [Listener at localhost.localdomain/34579] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-05 04:56:09,057 INFO [Listener at localhost.localdomain/34579] zookeeper.RecoverableZooKeeper(93): Process identifier=master:43797 connecting to ZooKeeper ensemble=127.0.0.1:56229 2023-04-05 04:56:09,095 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:437970x0, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-05 04:56:09,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:43797-0x1007f7bc4020000 connected 2023-04-05 04:56:09,122 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-05 04:56:09,123 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-05 04:56:09,127 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-05 04:56:09,134 DEBUG [Listener at localhost.localdomain/34579] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=43797 2023-04-05 04:56:09,135 DEBUG [Listener at localhost.localdomain/34579] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=43797 2023-04-05 04:56:09,135 DEBUG [Listener at localhost.localdomain/34579] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=43797 2023-04-05 04:56:09,136 DEBUG [Listener at localhost.localdomain/34579] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=43797 2023-04-05 04:56:09,136 DEBUG [Listener at localhost.localdomain/34579] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=43797 2023-04-05 04:56:09,142 INFO [Listener at localhost.localdomain/34579] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7, hbase.cluster.distributed=false 2023-04-05 04:56:09,207 INFO [Listener at localhost.localdomain/34579] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-04-05 04:56:09,208 INFO [Listener at localhost.localdomain/34579] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-05 04:56:09,208 INFO [Listener at localhost.localdomain/34579] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-05 04:56:09,208 INFO [Listener at localhost.localdomain/34579] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-05 04:56:09,209 INFO [Listener at localhost.localdomain/34579] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-05 04:56:09,209 INFO [Listener at localhost.localdomain/34579] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-05 04:56:09,216 INFO [Listener at localhost.localdomain/34579] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-05 04:56:09,231 INFO [Listener at localhost.localdomain/34579] ipc.NettyRpcServer(120): Bind to /148.251.75.209:38091 2023-04-05 04:56:09,233 INFO [Listener at localhost.localdomain/34579] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-05 04:56:09,239 DEBUG [Listener at localhost.localdomain/34579] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-05 04:56:09,240 INFO [Listener at localhost.localdomain/34579] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-05 04:56:09,243 INFO [Listener at localhost.localdomain/34579] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-05 04:56:09,244 INFO [Listener at localhost.localdomain/34579] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:38091 connecting to ZooKeeper ensemble=127.0.0.1:56229 2023-04-05 04:56:09,248 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:380910x0, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-05 04:56:09,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:38091-0x1007f7bc4020001 connected 2023-04-05 04:56:09,249 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-05 04:56:09,250 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-05 04:56:09,251 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-05 04:56:09,252 DEBUG [Listener at localhost.localdomain/34579] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=38091 2023-04-05 04:56:09,252 DEBUG [Listener at localhost.localdomain/34579] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=38091 2023-04-05 04:56:09,252 DEBUG [Listener at localhost.localdomain/34579] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=38091 2023-04-05 04:56:09,253 DEBUG [Listener at localhost.localdomain/34579] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=38091 2023-04-05 04:56:09,254 DEBUG [Listener at localhost.localdomain/34579] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=38091 2023-04-05 04:56:09,255 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase20.apache.org,43797,1680670568131 2023-04-05 04:56:09,263 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-05 04:56:09,265 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase20.apache.org,43797,1680670568131 2023-04-05 04:56:09,281 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-05 04:56:09,281 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-05 04:56:09,281 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-05 04:56:09,282 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-05 04:56:09,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-05 04:56:09,283 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase20.apache.org,43797,1680670568131 from backup master directory 2023-04-05 04:56:09,285 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase20.apache.org,43797,1680670568131 2023-04-05 04:56:09,285 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-05 04:56:09,286 WARN [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-05 04:56:09,286 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase20.apache.org,43797,1680670568131 2023-04-05 04:56:09,289 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-05 04:56:09,290 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-05 04:56:09,771 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/hbase.id with ID: 42ecdf6b-2073-474c-8e94-bf702b7d1253 2023-04-05 04:56:09,822 INFO [master/jenkins-hbase20:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-05 04:56:09,837 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-05 04:56:09,878 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x3eaf8ec6 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:56:09,908 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7d5804b3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:56:09,931 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-05 04:56:09,933 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-05 04:56:09,950 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-05 04:56:09,951 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-05 04:56:09,953 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-05 04:56:09,958 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-05 04:56:09,960 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-05 04:56:09,991 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/MasterData/data/master/store-tmp 2023-04-05 04:56:10,028 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 04:56:10,028 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-05 04:56:10,028 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-05 04:56:10,029 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-05 04:56:10,029 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-05 04:56:10,029 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-05 04:56:10,029 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-05 04:56:10,029 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-05 04:56:10,031 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/MasterData/WALs/jenkins-hbase20.apache.org,43797,1680670568131 2023-04-05 04:56:10,058 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C43797%2C1680670568131, suffix=, logDir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/MasterData/WALs/jenkins-hbase20.apache.org,43797,1680670568131, archiveDir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/MasterData/oldWALs, maxLogs=10 2023-04-05 04:56:10,120 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45635,DS-67667659-cf10-4833-b2f2-a62752fe1d31,DISK] 2023-04-05 04:56:10,131 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 04:56:10,210 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/MasterData/WALs/jenkins-hbase20.apache.org,43797,1680670568131/jenkins-hbase20.apache.org%2C43797%2C1680670568131.1680670570067 2023-04-05 04:56:10,211 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45635,DS-67667659-cf10-4833-b2f2-a62752fe1d31,DISK]] 2023-04-05 04:56:10,212 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-05 04:56:10,213 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 04:56:10,217 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-05 04:56:10,219 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-05 04:56:10,292 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-05 04:56:10,300 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-05 04:56:10,330 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-05 04:56:10,344 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 04:56:10,350 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-05 04:56:10,353 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-05 04:56:10,376 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-05 04:56:10,383 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-05 04:56:10,384 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59703522, jitterRate=-0.11034819483757019}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-05 04:56:10,384 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-05 04:56:10,386 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-05 04:56:10,409 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-05 04:56:10,410 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-05 04:56:10,412 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-05 04:56:10,413 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-05 04:56:10,444 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 30 msec 2023-04-05 04:56:10,444 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-05 04:56:10,475 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-05 04:56:10,482 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-05 04:56:10,504 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-05 04:56:10,507 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-05 04:56:10,509 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-05 04:56:10,514 INFO [master/jenkins-hbase20:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-05 04:56:10,517 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-05 04:56:10,520 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-05 04:56:10,521 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-05 04:56:10,521 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-05 04:56:10,535 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-05 04:56:10,539 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-05 04:56:10,539 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-05 04:56:10,539 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-05 04:56:10,540 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase20.apache.org,43797,1680670568131, sessionid=0x1007f7bc4020000, setting cluster-up flag (Was=false) 2023-04-05 04:56:10,553 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-05 04:56:10,557 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-05 04:56:10,558 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,43797,1680670568131 2023-04-05 04:56:10,562 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-05 04:56:10,566 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-05 04:56:10,568 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,43797,1680670568131 2023-04-05 04:56:10,570 WARN [master/jenkins-hbase20:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.hbase-snapshot/.tmp 2023-04-05 04:56:10,658 INFO [RS:0;jenkins-hbase20:38091] regionserver.HRegionServer(952): ClusterId : 42ecdf6b-2073-474c-8e94-bf702b7d1253 2023-04-05 04:56:10,661 DEBUG [RS:0;jenkins-hbase20:38091] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-05 04:56:10,667 DEBUG [RS:0;jenkins-hbase20:38091] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-05 04:56:10,667 DEBUG [RS:0;jenkins-hbase20:38091] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-05 04:56:10,669 DEBUG [RS:0;jenkins-hbase20:38091] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-05 04:56:10,671 DEBUG [RS:0;jenkins-hbase20:38091] zookeeper.ReadOnlyZKClient(139): Connect 0x0c53079d to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:56:10,686 DEBUG [RS:0;jenkins-hbase20:38091] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@294f867, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:56:10,686 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-05 04:56:10,687 DEBUG [RS:0;jenkins-hbase20:38091] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6ab6ced, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-04-05 04:56:10,697 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-04-05 04:56:10,698 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-04-05 04:56:10,698 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-04-05 04:56:10,698 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-04-05 04:56:10,698 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase20:0, corePoolSize=10, maxPoolSize=10 2023-04-05 04:56:10,699 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-05 04:56:10,699 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-04-05 04:56:10,699 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-05 04:56:10,701 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1680670600700 2023-04-05 04:56:10,702 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-05 04:56:10,708 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-05 04:56:10,709 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-05 04:56:10,713 DEBUG [RS:0;jenkins-hbase20:38091] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase20:38091 2023-04-05 04:56:10,714 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-05 04:56:10,714 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-05 04:56:10,718 INFO [RS:0;jenkins-hbase20:38091] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-05 04:56:10,718 INFO [RS:0;jenkins-hbase20:38091] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-05 04:56:10,719 DEBUG [RS:0;jenkins-hbase20:38091] regionserver.HRegionServer(1023): About to register with Master. 2023-04-05 04:56:10,722 INFO [RS:0;jenkins-hbase20:38091] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase20.apache.org,43797,1680670568131 with isa=jenkins-hbase20.apache.org/148.251.75.209:38091, startcode=1680670569206 2023-04-05 04:56:10,723 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-05 04:56:10,724 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-05 04:56:10,725 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-05 04:56:10,725 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-05 04:56:10,726 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-05 04:56:10,727 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-05 04:56:10,729 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-05 04:56:10,729 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-05 04:56:10,732 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-05 04:56:10,732 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-05 04:56:10,735 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1680670570734,5,FailOnTimeoutGroup] 2023-04-05 04:56:10,735 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1680670570735,5,FailOnTimeoutGroup] 2023-04-05 04:56:10,735 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-05 04:56:10,735 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-05 04:56:10,736 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-05 04:56:10,737 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-05 04:56:10,739 DEBUG [RS:0;jenkins-hbase20:38091] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-05 04:56:10,792 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:33643, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-05 04:56:10,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:10,821 DEBUG [RS:0;jenkins-hbase20:38091] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7 2023-04-05 04:56:10,821 DEBUG [RS:0;jenkins-hbase20:38091] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost.localdomain:39291 2023-04-05 04:56:10,821 DEBUG [RS:0;jenkins-hbase20:38091] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-05 04:56:10,827 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-05 04:56:10,827 DEBUG [RS:0;jenkins-hbase20:38091] zookeeper.ZKUtil(162): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:10,828 WARN [RS:0;jenkins-hbase20:38091] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-05 04:56:10,828 INFO [RS:0;jenkins-hbase20:38091] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-05 04:56:10,829 DEBUG [RS:0;jenkins-hbase20:38091] regionserver.HRegionServer(1947): logDir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/WALs/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:10,829 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,38091,1680670569206] 2023-04-05 04:56:10,837 DEBUG [RS:0;jenkins-hbase20:38091] zookeeper.ZKUtil(162): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:10,847 DEBUG [RS:0;jenkins-hbase20:38091] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-05 04:56:10,856 INFO [RS:0;jenkins-hbase20:38091] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-05 04:56:10,878 INFO [RS:0;jenkins-hbase20:38091] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-05 04:56:10,882 INFO [RS:0;jenkins-hbase20:38091] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-05 04:56:10,883 INFO [RS:0;jenkins-hbase20:38091] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-05 04:56:10,885 INFO [RS:0;jenkins-hbase20:38091] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-05 04:56:10,891 INFO [RS:0;jenkins-hbase20:38091] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-05 04:56:10,892 DEBUG [RS:0;jenkins-hbase20:38091] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-05 04:56:10,892 DEBUG [RS:0;jenkins-hbase20:38091] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-05 04:56:10,892 DEBUG [RS:0;jenkins-hbase20:38091] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-05 04:56:10,892 DEBUG [RS:0;jenkins-hbase20:38091] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-05 04:56:10,893 DEBUG [RS:0;jenkins-hbase20:38091] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-05 04:56:10,893 DEBUG [RS:0;jenkins-hbase20:38091] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-04-05 04:56:10,893 DEBUG [RS:0;jenkins-hbase20:38091] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-05 04:56:10,893 DEBUG [RS:0;jenkins-hbase20:38091] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-05 04:56:10,893 DEBUG [RS:0;jenkins-hbase20:38091] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-05 04:56:10,893 DEBUG [RS:0;jenkins-hbase20:38091] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-05 04:56:10,894 INFO [RS:0;jenkins-hbase20:38091] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-05 04:56:10,894 INFO [RS:0;jenkins-hbase20:38091] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-05 04:56:10,894 INFO [RS:0;jenkins-hbase20:38091] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-05 04:56:10,907 INFO [RS:0;jenkins-hbase20:38091] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-05 04:56:10,909 INFO [RS:0;jenkins-hbase20:38091] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,38091,1680670569206-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-05 04:56:10,923 INFO [RS:0;jenkins-hbase20:38091] regionserver.Replication(203): jenkins-hbase20.apache.org,38091,1680670569206 started 2023-04-05 04:56:10,923 INFO [RS:0;jenkins-hbase20:38091] regionserver.HRegionServer(1638): Serving as jenkins-hbase20.apache.org,38091,1680670569206, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:38091, sessionid=0x1007f7bc4020001 2023-04-05 04:56:10,923 DEBUG [RS:0;jenkins-hbase20:38091] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-05 04:56:10,923 DEBUG [RS:0;jenkins-hbase20:38091] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:10,923 DEBUG [RS:0;jenkins-hbase20:38091] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,38091,1680670569206' 2023-04-05 04:56:10,923 DEBUG [RS:0;jenkins-hbase20:38091] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:10,924 DEBUG [RS:0;jenkins-hbase20:38091] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:10,925 DEBUG [RS:0;jenkins-hbase20:38091] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-05 04:56:10,925 DEBUG [RS:0;jenkins-hbase20:38091] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-05 04:56:10,925 DEBUG [RS:0;jenkins-hbase20:38091] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:10,925 DEBUG [RS:0;jenkins-hbase20:38091] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,38091,1680670569206' 2023-04-05 04:56:10,925 DEBUG [RS:0;jenkins-hbase20:38091] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-05 04:56:10,925 DEBUG [RS:0;jenkins-hbase20:38091] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-05 04:56:10,926 DEBUG [RS:0;jenkins-hbase20:38091] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-05 04:56:10,926 INFO [RS:0;jenkins-hbase20:38091] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-05 04:56:10,926 INFO [RS:0;jenkins-hbase20:38091] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-05 04:56:11,037 INFO [RS:0;jenkins-hbase20:38091] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C38091%2C1680670569206, suffix=, logDir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/WALs/jenkins-hbase20.apache.org,38091,1680670569206, archiveDir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/oldWALs, maxLogs=32 2023-04-05 04:56:11,053 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45635,DS-67667659-cf10-4833-b2f2-a62752fe1d31,DISK] 2023-04-05 04:56:11,056 INFO [RS:0;jenkins-hbase20:38091] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/WALs/jenkins-hbase20.apache.org,38091,1680670569206/jenkins-hbase20.apache.org%2C38091%2C1680670569206.1680670571040 2023-04-05 04:56:11,056 DEBUG [RS:0;jenkins-hbase20:38091] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45635,DS-67667659-cf10-4833-b2f2-a62752fe1d31,DISK]] 2023-04-05 04:56:11,152 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-05 04:56:11,153 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-05 04:56:11,153 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7 2023-04-05 04:56:11,166 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 04:56:11,168 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-05 04:56:11,171 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/hbase/meta/1588230740/info 2023-04-05 04:56:11,171 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-05 04:56:11,173 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 04:56:11,173 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-05 04:56:11,176 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/hbase/meta/1588230740/rep_barrier 2023-04-05 04:56:11,177 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-05 04:56:11,177 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 04:56:11,178 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-05 04:56:11,180 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/hbase/meta/1588230740/table 2023-04-05 04:56:11,180 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-05 04:56:11,181 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 04:56:11,183 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/hbase/meta/1588230740 2023-04-05 04:56:11,184 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/hbase/meta/1588230740 2023-04-05 04:56:11,187 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-05 04:56:11,188 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-05 04:56:11,191 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-05 04:56:11,192 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61886956, jitterRate=-0.07781249284744263}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-05 04:56:11,192 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-05 04:56:11,192 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-05 04:56:11,192 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-05 04:56:11,192 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-05 04:56:11,192 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-05 04:56:11,192 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-05 04:56:11,194 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-05 04:56:11,194 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-05 04:56:11,199 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-05 04:56:11,199 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-05 04:56:11,206 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-05 04:56:11,218 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-05 04:56:11,220 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-05 04:56:11,372 DEBUG [jenkins-hbase20:43797] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-05 04:56:11,375 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,38091,1680670569206, state=OPENING 2023-04-05 04:56:11,381 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-05 04:56:11,382 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-05 04:56:11,382 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-05 04:56:11,384 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,38091,1680670569206}] 2023-04-05 04:56:11,566 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:11,568 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-05 04:56:11,572 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49048, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-05 04:56:11,582 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-05 04:56:11,582 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-05 04:56:11,586 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C38091%2C1680670569206.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/WALs/jenkins-hbase20.apache.org,38091,1680670569206, archiveDir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/oldWALs, maxLogs=32 2023-04-05 04:56:11,603 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45635,DS-67667659-cf10-4833-b2f2-a62752fe1d31,DISK] 2023-04-05 04:56:11,607 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/WALs/jenkins-hbase20.apache.org,38091,1680670569206/jenkins-hbase20.apache.org%2C38091%2C1680670569206.meta.1680670571588.meta 2023-04-05 04:56:11,607 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45635,DS-67667659-cf10-4833-b2f2-a62752fe1d31,DISK]] 2023-04-05 04:56:11,608 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-05 04:56:11,609 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-05 04:56:11,622 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-05 04:56:11,626 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-05 04:56:11,630 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-05 04:56:11,630 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 04:56:11,630 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-05 04:56:11,630 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-05 04:56:11,633 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-05 04:56:11,635 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/hbase/meta/1588230740/info 2023-04-05 04:56:11,635 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/hbase/meta/1588230740/info 2023-04-05 04:56:11,635 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-05 04:56:11,636 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 04:56:11,636 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-05 04:56:11,638 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/hbase/meta/1588230740/rep_barrier 2023-04-05 04:56:11,638 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/hbase/meta/1588230740/rep_barrier 2023-04-05 04:56:11,638 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-05 04:56:11,639 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 04:56:11,639 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-05 04:56:11,641 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/hbase/meta/1588230740/table 2023-04-05 04:56:11,641 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/hbase/meta/1588230740/table 2023-04-05 04:56:11,641 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-05 04:56:11,642 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 04:56:11,644 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/hbase/meta/1588230740 2023-04-05 04:56:11,646 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/hbase/meta/1588230740 2023-04-05 04:56:11,649 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-05 04:56:11,652 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-05 04:56:11,653 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70225185, jitterRate=0.04643680155277252}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-05 04:56:11,653 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-05 04:56:11,661 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1680670571558 2023-04-05 04:56:11,679 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-05 04:56:11,680 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-05 04:56:11,680 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,38091,1680670569206, state=OPEN 2023-04-05 04:56:11,682 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-05 04:56:11,682 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-05 04:56:11,687 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-05 04:56:11,687 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,38091,1680670569206 in 298 msec 2023-04-05 04:56:11,692 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-05 04:56:11,692 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 482 msec 2023-04-05 04:56:11,700 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.0830 sec 2023-04-05 04:56:11,700 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1680670571700, completionTime=-1 2023-04-05 04:56:11,701 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-05 04:56:11,701 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-05 04:56:11,759 DEBUG [hconnection-0x1dc5d24e-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:56:11,762 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49054, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:56:11,777 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-05 04:56:11,777 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1680670631777 2023-04-05 04:56:11,777 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1680670691777 2023-04-05 04:56:11,777 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 76 msec 2023-04-05 04:56:11,804 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,43797,1680670568131-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-05 04:56:11,804 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,43797,1680670568131-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-05 04:56:11,805 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,43797,1680670568131-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-05 04:56:11,807 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase20:43797, period=300000, unit=MILLISECONDS is enabled. 2023-04-05 04:56:11,807 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-05 04:56:11,813 DEBUG [master/jenkins-hbase20:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-05 04:56:11,821 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-05 04:56:11,822 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-05 04:56:11,832 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-05 04:56:11,834 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-05 04:56:11,838 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-05 04:56:11,857 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/hbase/namespace/557b26e17f78141e7d0cde2ccb5b79b0 2023-04-05 04:56:11,859 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/hbase/namespace/557b26e17f78141e7d0cde2ccb5b79b0 empty. 2023-04-05 04:56:11,860 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/hbase/namespace/557b26e17f78141e7d0cde2ccb5b79b0 2023-04-05 04:56:11,860 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-05 04:56:11,883 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-05 04:56:11,885 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 557b26e17f78141e7d0cde2ccb5b79b0, NAME => 'hbase:namespace,,1680670571822.557b26e17f78141e7d0cde2ccb5b79b0.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp 2023-04-05 04:56:12,298 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680670571822.557b26e17f78141e7d0cde2ccb5b79b0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 04:56:12,298 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 557b26e17f78141e7d0cde2ccb5b79b0, disabling compactions & flushes 2023-04-05 04:56:12,298 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680670571822.557b26e17f78141e7d0cde2ccb5b79b0. 2023-04-05 04:56:12,298 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680670571822.557b26e17f78141e7d0cde2ccb5b79b0. 2023-04-05 04:56:12,298 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680670571822.557b26e17f78141e7d0cde2ccb5b79b0. after waiting 0 ms 2023-04-05 04:56:12,298 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680670571822.557b26e17f78141e7d0cde2ccb5b79b0. 2023-04-05 04:56:12,298 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680670571822.557b26e17f78141e7d0cde2ccb5b79b0. 2023-04-05 04:56:12,298 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 557b26e17f78141e7d0cde2ccb5b79b0: 2023-04-05 04:56:12,303 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-05 04:56:12,317 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1680670571822.557b26e17f78141e7d0cde2ccb5b79b0.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680670572306"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680670572306"}]},"ts":"1680670572306"} 2023-04-05 04:56:12,345 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-05 04:56:12,347 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-05 04:56:12,351 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680670572347"}]},"ts":"1680670572347"} 2023-04-05 04:56:12,355 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-05 04:56:12,362 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=557b26e17f78141e7d0cde2ccb5b79b0, ASSIGN}] 2023-04-05 04:56:12,365 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=557b26e17f78141e7d0cde2ccb5b79b0, ASSIGN 2023-04-05 04:56:12,366 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=557b26e17f78141e7d0cde2ccb5b79b0, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,38091,1680670569206; forceNewPlan=false, retain=false 2023-04-05 04:56:12,518 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=557b26e17f78141e7d0cde2ccb5b79b0, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:12,518 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1680670571822.557b26e17f78141e7d0cde2ccb5b79b0.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680670572517"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680670572517"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680670572517"}]},"ts":"1680670572517"} 2023-04-05 04:56:12,522 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 557b26e17f78141e7d0cde2ccb5b79b0, server=jenkins-hbase20.apache.org,38091,1680670569206}] 2023-04-05 04:56:12,681 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1680670571822.557b26e17f78141e7d0cde2ccb5b79b0. 2023-04-05 04:56:12,682 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 557b26e17f78141e7d0cde2ccb5b79b0, NAME => 'hbase:namespace,,1680670571822.557b26e17f78141e7d0cde2ccb5b79b0.', STARTKEY => '', ENDKEY => ''} 2023-04-05 04:56:12,683 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 557b26e17f78141e7d0cde2ccb5b79b0 2023-04-05 04:56:12,683 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680670571822.557b26e17f78141e7d0cde2ccb5b79b0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 04:56:12,683 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 557b26e17f78141e7d0cde2ccb5b79b0 2023-04-05 04:56:12,683 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 557b26e17f78141e7d0cde2ccb5b79b0 2023-04-05 04:56:12,685 INFO [StoreOpener-557b26e17f78141e7d0cde2ccb5b79b0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 557b26e17f78141e7d0cde2ccb5b79b0 2023-04-05 04:56:12,687 DEBUG [StoreOpener-557b26e17f78141e7d0cde2ccb5b79b0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/hbase/namespace/557b26e17f78141e7d0cde2ccb5b79b0/info 2023-04-05 04:56:12,687 DEBUG [StoreOpener-557b26e17f78141e7d0cde2ccb5b79b0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/hbase/namespace/557b26e17f78141e7d0cde2ccb5b79b0/info 2023-04-05 04:56:12,688 INFO [StoreOpener-557b26e17f78141e7d0cde2ccb5b79b0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 557b26e17f78141e7d0cde2ccb5b79b0 columnFamilyName info 2023-04-05 04:56:12,689 INFO [StoreOpener-557b26e17f78141e7d0cde2ccb5b79b0-1] regionserver.HStore(310): Store=557b26e17f78141e7d0cde2ccb5b79b0/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 04:56:12,690 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/hbase/namespace/557b26e17f78141e7d0cde2ccb5b79b0 2023-04-05 04:56:12,691 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/hbase/namespace/557b26e17f78141e7d0cde2ccb5b79b0 2023-04-05 04:56:12,695 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 557b26e17f78141e7d0cde2ccb5b79b0 2023-04-05 04:56:12,698 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/hbase/namespace/557b26e17f78141e7d0cde2ccb5b79b0/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-05 04:56:12,699 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 557b26e17f78141e7d0cde2ccb5b79b0; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64953359, jitterRate=-0.03211952745914459}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-05 04:56:12,699 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 557b26e17f78141e7d0cde2ccb5b79b0: 2023-04-05 04:56:12,701 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1680670571822.557b26e17f78141e7d0cde2ccb5b79b0., pid=6, masterSystemTime=1680670572675 2023-04-05 04:56:12,704 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1680670571822.557b26e17f78141e7d0cde2ccb5b79b0. 2023-04-05 04:56:12,705 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1680670571822.557b26e17f78141e7d0cde2ccb5b79b0. 2023-04-05 04:56:12,706 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=557b26e17f78141e7d0cde2ccb5b79b0, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:12,706 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1680670571822.557b26e17f78141e7d0cde2ccb5b79b0.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680670572705"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680670572705"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680670572705"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680670572705"}]},"ts":"1680670572705"} 2023-04-05 04:56:12,714 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-05 04:56:12,714 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 557b26e17f78141e7d0cde2ccb5b79b0, server=jenkins-hbase20.apache.org,38091,1680670569206 in 188 msec 2023-04-05 04:56:12,717 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-05 04:56:12,717 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=557b26e17f78141e7d0cde2ccb5b79b0, ASSIGN in 352 msec 2023-04-05 04:56:12,719 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-05 04:56:12,719 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680670572719"}]},"ts":"1680670572719"} 2023-04-05 04:56:12,722 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-05 04:56:12,725 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-05 04:56:12,728 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 901 msec 2023-04-05 04:56:12,735 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-05 04:56:12,743 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-05 04:56:12,743 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-05 04:56:12,780 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-05 04:56:12,796 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-05 04:56:12,803 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 31 msec 2023-04-05 04:56:12,815 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-05 04:56:12,827 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-05 04:56:12,832 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 17 msec 2023-04-05 04:56:12,846 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-05 04:56:12,848 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-05 04:56:12,848 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.562sec 2023-04-05 04:56:12,850 INFO [master/jenkins-hbase20:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-05 04:56:12,852 INFO [master/jenkins-hbase20:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-05 04:56:12,852 INFO [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-05 04:56:12,853 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,43797,1680670568131-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-05 04:56:12,854 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,43797,1680670568131-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-05 04:56:12,864 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x70531455 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:56:12,865 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-05 04:56:12,868 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a972b34, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:56:12,884 DEBUG [hconnection-0x5e53e335-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:56:12,897 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49066, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:56:12,911 INFO [Listener at localhost.localdomain/34579] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase20.apache.org,43797,1680670568131 2023-04-05 04:56:12,933 INFO [Listener at localhost.localdomain/34579] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=207, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=188, ProcessCount=167, AvailableMemoryMB=7049 2023-04-05 04:56:12,948 DEBUG [Listener at localhost.localdomain/34579] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-05 04:56:12,951 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52812, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-05 04:56:12,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-05 04:56:12,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-05 04:56:12,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-05 04:56:12,967 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-05 04:56:12,969 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-05 04:56:12,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-05 04:56:12,972 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:12,974 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7 empty. 2023-04-05 04:56:12,974 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:12,974 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-05 04:56:12,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-05 04:56:13,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-05 04:56:13,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-05 04:56:13,393 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-05 04:56:13,396 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => ed7bbafaddb97066dca4b03f35c9eee7, NAME => 'TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp 2023-04-05 04:56:13,420 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 04:56:13,420 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing ed7bbafaddb97066dca4b03f35c9eee7, disabling compactions & flushes 2023-04-05 04:56:13,420 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:13,420 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:13,420 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. after waiting 0 ms 2023-04-05 04:56:13,420 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:13,420 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:13,420 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:13,426 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-05 04:56:13,428 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670573427"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680670573427"}]},"ts":"1680670573427"} 2023-04-05 04:56:13,431 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-05 04:56:13,432 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-05 04:56:13,433 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680670573432"}]},"ts":"1680670573432"} 2023-04-05 04:56:13,435 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-05 04:56:13,438 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ed7bbafaddb97066dca4b03f35c9eee7, ASSIGN}] 2023-04-05 04:56:13,441 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ed7bbafaddb97066dca4b03f35c9eee7, ASSIGN 2023-04-05 04:56:13,443 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=ed7bbafaddb97066dca4b03f35c9eee7, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,38091,1680670569206; forceNewPlan=false, retain=false 2023-04-05 04:56:13,594 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=ed7bbafaddb97066dca4b03f35c9eee7, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:13,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-05 04:56:13,595 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670573594"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680670573594"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680670573594"}]},"ts":"1680670573594"} 2023-04-05 04:56:13,599 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206}] 2023-04-05 04:56:13,760 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:13,760 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ed7bbafaddb97066dca4b03f35c9eee7, NAME => 'TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.', STARTKEY => '', ENDKEY => ''} 2023-04-05 04:56:13,761 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:13,761 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 04:56:13,761 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:13,761 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:13,763 INFO [StoreOpener-ed7bbafaddb97066dca4b03f35c9eee7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:13,766 DEBUG [StoreOpener-ed7bbafaddb97066dca4b03f35c9eee7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A 2023-04-05 04:56:13,766 DEBUG [StoreOpener-ed7bbafaddb97066dca4b03f35c9eee7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A 2023-04-05 04:56:13,768 INFO [StoreOpener-ed7bbafaddb97066dca4b03f35c9eee7-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 04:56:13,768 INFO [StoreOpener-ed7bbafaddb97066dca4b03f35c9eee7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ed7bbafaddb97066dca4b03f35c9eee7 columnFamilyName A 2023-04-05 04:56:13,769 INFO [StoreOpener-ed7bbafaddb97066dca4b03f35c9eee7-1] regionserver.HStore(310): Store=ed7bbafaddb97066dca4b03f35c9eee7/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 04:56:13,770 INFO [StoreOpener-ed7bbafaddb97066dca4b03f35c9eee7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:13,772 DEBUG [StoreOpener-ed7bbafaddb97066dca4b03f35c9eee7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B 2023-04-05 04:56:13,772 DEBUG [StoreOpener-ed7bbafaddb97066dca4b03f35c9eee7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B 2023-04-05 04:56:13,773 INFO [StoreOpener-ed7bbafaddb97066dca4b03f35c9eee7-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 04:56:13,773 INFO [StoreOpener-ed7bbafaddb97066dca4b03f35c9eee7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ed7bbafaddb97066dca4b03f35c9eee7 columnFamilyName B 2023-04-05 04:56:13,774 INFO [StoreOpener-ed7bbafaddb97066dca4b03f35c9eee7-1] regionserver.HStore(310): Store=ed7bbafaddb97066dca4b03f35c9eee7/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 04:56:13,774 INFO [StoreOpener-ed7bbafaddb97066dca4b03f35c9eee7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:13,779 DEBUG [StoreOpener-ed7bbafaddb97066dca4b03f35c9eee7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C 2023-04-05 04:56:13,779 DEBUG [StoreOpener-ed7bbafaddb97066dca4b03f35c9eee7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C 2023-04-05 04:56:13,779 INFO [StoreOpener-ed7bbafaddb97066dca4b03f35c9eee7-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 04:56:13,779 INFO [StoreOpener-ed7bbafaddb97066dca4b03f35c9eee7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ed7bbafaddb97066dca4b03f35c9eee7 columnFamilyName C 2023-04-05 04:56:13,780 INFO [StoreOpener-ed7bbafaddb97066dca4b03f35c9eee7-1] regionserver.HStore(310): Store=ed7bbafaddb97066dca4b03f35c9eee7/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 04:56:13,781 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:13,784 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:13,785 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:13,789 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-05 04:56:13,792 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:13,797 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-05 04:56:13,798 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened ed7bbafaddb97066dca4b03f35c9eee7; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61289178, jitterRate=-0.08672007918357849}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-05 04:56:13,798 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:13,799 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7., pid=11, masterSystemTime=1680670573753 2023-04-05 04:56:13,803 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:13,803 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:13,804 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=ed7bbafaddb97066dca4b03f35c9eee7, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:13,805 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670573804"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680670573804"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680670573804"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680670573804"}]},"ts":"1680670573804"} 2023-04-05 04:56:13,812 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-05 04:56:13,812 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 in 209 msec 2023-04-05 04:56:13,816 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-05 04:56:13,817 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=ed7bbafaddb97066dca4b03f35c9eee7, ASSIGN in 374 msec 2023-04-05 04:56:13,818 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-05 04:56:13,819 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680670573819"}]},"ts":"1680670573819"} 2023-04-05 04:56:13,821 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-05 04:56:13,824 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-05 04:56:13,827 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 861 msec 2023-04-05 04:56:14,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-05 04:56:14,099 INFO [Listener at localhost.localdomain/34579] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-05 04:56:14,110 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x12582a26 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:56:14,115 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@458f9a9e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:56:14,119 DEBUG [Listener at localhost.localdomain/34579] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:56:14,124 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49072, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:56:14,130 DEBUG [Listener at localhost.localdomain/34579] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-05 04:56:14,134 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52818, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-05 04:56:14,143 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x3e0ba002 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:56:14,147 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@20e4e4f3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:56:14,148 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x77443fb0 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:56:14,163 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13e837e0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:56:14,164 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x136956fd to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:56:14,171 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6362e5a0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:56:14,172 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x4654132d to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:56:14,186 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6be8c20b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:56:14,187 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x00e18ba3 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:56:14,197 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@46b5e40, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:56:14,198 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x1439cba9 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:56:14,211 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5fe050a3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:56:14,211 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x3fed686f to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:56:14,242 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@655398b8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:56:14,243 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x6a7ef79d to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:56:14,247 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13112432, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:56:14,247 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x6ab9467e to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:56:14,255 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@776686ab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:56:14,271 DEBUG [hconnection-0x1440f91b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:56:14,278 DEBUG [hconnection-0x6529cc2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:56:14,278 DEBUG [hconnection-0x3c8bee42-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:56:14,278 DEBUG [hconnection-0x70df3ad-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:56:14,287 DEBUG [hconnection-0x6d035815-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:56:14,290 DEBUG [hconnection-0x2da18d42-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:56:14,294 DEBUG [hconnection-0x53790481-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:56:14,295 DEBUG [hconnection-0x43fe1fb2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:56:14,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:14,302 DEBUG [hconnection-0x5559b7c4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:56:14,303 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49098, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:56:14,310 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49082, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:56:14,310 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49112, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:56:14,313 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49110, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:56:14,315 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49134, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:56:14,316 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49118, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:56:14,318 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49116, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:56:14,324 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49148, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:56:14,327 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49154, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:56:14,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:14,330 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:14,330 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:14,331 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:14,331 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:14,332 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:14,332 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:14,333 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:14,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:14,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:14,334 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:14,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:14,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:14,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:14,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:14,335 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:14,336 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:14,336 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:14,346 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:14,373 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:14,373 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:14,382 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:14,384 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:14,384 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:14,390 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:14,393 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:14,394 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ed7bbafaddb97066dca4b03f35c9eee7 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 04:56:14,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:14,397 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:14,406 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=A 2023-04-05 04:56:14,408 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:14,408 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=B 2023-04-05 04:56:14,409 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:14,409 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=C 2023-04-05 04:56:14,409 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:14,550 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:14,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49098 deadline: 1680670634516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:14,560 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:14,562 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:14,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670634541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:14,563 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:14,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670634546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:14,547 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680670574467, max timestamp=1680670574521Num uniques 859; 2023-04-05 04:56:14,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670634524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:14,556 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680670574467, max timestamp=1680670574528Num uniques 859; 2023-04-05 04:56:14,547 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670574467, max timestamp=1680670574521Num uniques 859; 2023-04-05 04:56:14,566 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-05 04:56:14,565 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-05 04:56:14,566 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:56:14,665 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/430c0819709b4c71aa086159941e048e 2023-04-05 04:56:14,730 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:14,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49098 deadline: 1680670634722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:14,732 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:14,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670634723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:14,733 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:14,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670634726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:14,734 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:14,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49118 deadline: 1680670634727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:14,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:14,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670634727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:14,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:14,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49098 deadline: 1680670634835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:14,838 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:14,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670634836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:14,839 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:14,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49118 deadline: 1680670634838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:14,938 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:14,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670634937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:14,944 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/046d7cbf21584d27ae1ec2fd07dfe460 2023-04-05 04:56:14,950 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:14,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670634948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,044 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:15,045 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:15,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49098 deadline: 1680670635042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670635042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,059 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:15,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49118 deadline: 1680670635047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,247 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:15,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670635244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,257 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:15,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670635254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,353 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:15,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670635352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,357 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:15,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49098 deadline: 1680670635353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,373 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:15,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49118 deadline: 1680670635371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,462 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/4c4cd0a96b4f46178a224a8ab1828bd8 2023-04-05 04:56:15,475 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/430c0819709b4c71aa086159941e048e as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/430c0819709b4c71aa086159941e048e 2023-04-05 04:56:15,488 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/430c0819709b4c71aa086159941e048e, entries=200, sequenceid=11, filesize=13.9 K 2023-04-05 04:56:15,498 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/046d7cbf21584d27ae1ec2fd07dfe460 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/046d7cbf21584d27ae1ec2fd07dfe460 2023-04-05 04:56:15,516 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/046d7cbf21584d27ae1ec2fd07dfe460, entries=150, sequenceid=11, filesize=11.6 K 2023-04-05 04:56:15,518 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/4c4cd0a96b4f46178a224a8ab1828bd8 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/4c4cd0a96b4f46178a224a8ab1828bd8 2023-04-05 04:56:15,543 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/4c4cd0a96b4f46178a224a8ab1828bd8, entries=150, sequenceid=11, filesize=11.6 K 2023-04-05 04:56:15,548 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=133.39 KB/136590 for ed7bbafaddb97066dca4b03f35c9eee7 in 1153ms, sequenceid=11, compaction requested=false 2023-04-05 04:56:15,556 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-05 04:56:15,559 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:15,559 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:15,560 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:15,562 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:15,562 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,563 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:15,563 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:15,567 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:15,567 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:15,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:15,568 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:15,568 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:15,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:15,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:15,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:15,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:15,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:15,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14368008[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:15,571 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:15,571 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:15,579 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:15,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:15,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:15,579 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:15,579 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:15,579 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,580 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:15,580 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:15,586 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,586 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:15,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:15,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:15,586 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:15,587 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:15,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:15,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:15,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:15,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:15,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:15,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:15,598 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:15,598 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:15,598 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:15,598 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:15,598 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:15,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:15,603 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:15,603 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:15,603 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:15,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:15,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:15,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:15,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:15,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:15,608 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:15,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:15,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:15,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:15,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:15,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:15,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:15,611 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:15,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:15,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,628 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:15,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:15,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:15,628 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,629 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:15,629 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:15,629 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,629 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:15,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:15,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:15,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:15,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:15,629 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:15,629 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:15,637 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:15,637 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:15,637 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:15,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:15,637 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:15,738 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:15,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:15,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:15,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:15,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:15,774 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:15,774 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:15,779 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:15,779 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:15,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:15,780 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ed7bbafaddb97066dca4b03f35c9eee7 3/3 column families, dataSize=146.81 KB heapSize=423.70 KB 2023-04-05 04:56:15,780 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=A 2023-04-05 04:56:15,781 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-05 04:56:15,781 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=B 2023-04-05 04:56:15,781 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-05 04:56:15,781 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=C 2023-04-05 04:56:15,781 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-05 04:56:15,783 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:15,784 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:15,785 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:15,785 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:15,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:15,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:15,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:15,801 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,801 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:15,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:15,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:15,802 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:15,810 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:15,810 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:15,810 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:15,811 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:15,814 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:15,825 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:15,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670635808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,837 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:15,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670635809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,864 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:15,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49098 deadline: 1680670635862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,865 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:15,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670635863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,880 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:15,887 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:15,894 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:15,894 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:15,894 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:15,887 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:15,897 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:15,897 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:15,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49118 deadline: 1680670635877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,898 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:15,898 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,898 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:15,898 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:15,912 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,912 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:15,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:15,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:15,913 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:15,913 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:15,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:15,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:15,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:15,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b067b99[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:15,915 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:15,915 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:15,918 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:15,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:15,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:15,919 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:15,919 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:15,919 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:15,919 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,919 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:15,933 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:15,933 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:15,933 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:15,935 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:15,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:15,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:15,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:15,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:15,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:15,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:15,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:15,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:15,940 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:15,940 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:15,940 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:15,940 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:15,940 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:15,944 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:15,945 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:15,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:15,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:15,944 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:15,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:15,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:15,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:15,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:15,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:15,945 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:15,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:15,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:15,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:15,948 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:15,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:15,950 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:15,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670635939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,951 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:15,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:15,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670635939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,962 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:15,962 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:15,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:15,962 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:15,962 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,967 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:15,967 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:15,967 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:15,967 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:15,967 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:15,967 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:15,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:15,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:15,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:15,968 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:15,968 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:15,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:15,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:15,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:16,068 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:16,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:16,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:16,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:16,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:16,078 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:16,078 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:16,083 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:16,083 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:16,084 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:16,084 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:16,085 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:16,085 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:16,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:16,086 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,086 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:16,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:16,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:16,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:16,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:16,087 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:16,087 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:16,087 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:16,087 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:16,088 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:16,088 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:16,091 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:16,091 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:16,095 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:16,100 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:16,100 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:16,101 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:16,105 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:16,105 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:16,105 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,105 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:16,105 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:16,108 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:16,109 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:16,109 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:16,111 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:16,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:16,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:16,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:16,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:16,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:16,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:16,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b01b8b7[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:16,115 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:16,115 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:16,121 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:16,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:16,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:16,121 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:16,121 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:16,121 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:16,121 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,122 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:16,135 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:16,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:16,136 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:16,136 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:16,136 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:16,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:16,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:16,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:16,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:16,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:16,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:16,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:16,147 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:16,147 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:16,147 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:16,147 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:16,147 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:16,149 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:16,149 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:16,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:16,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:16,149 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:16,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:16,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:16,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:16,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:16,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:16,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:16,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:16,151 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:16,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:16,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:16,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,154 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:16,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:16,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,163 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,163 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:16,163 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:16,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:16,163 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:16,163 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:16,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:16,166 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:16,166 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,166 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:16,166 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:16,166 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:16,168 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:16,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670636154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:16,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:16,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:16,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:16,172 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:16,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:16,172 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:16,173 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:16,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670636162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,272 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:16,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:16,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:16,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:16,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:16,298 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:16,298 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:16,299 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:16,300 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:16,311 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:16,311 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:16,311 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/c13d76fee5d1496aa3ca21822f2f5a1b 2023-04-05 04:56:16,312 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:16,312 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:16,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:16,313 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,313 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:16,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:16,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:16,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:16,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:16,317 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:16,330 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:16,333 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:16,333 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:16,333 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:16,335 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:16,338 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:16,338 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:16,340 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:16,340 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:16,340 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:16,340 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:16,351 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:16,351 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:16,351 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,351 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:16,351 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:16,364 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:16,364 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:16,364 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:16,368 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:16,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:16,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:16,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:16,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:16,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,385 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=48.80 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/177f9cdd9dff4ad081e3409229f4ff64 2023-04-05 04:56:16,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:16,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:16,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ec32707[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:16,387 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:16,387 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:16,388 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:16,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:16,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:16,388 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:16,389 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:16,389 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:16,392 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:16,392 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,392 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:16,392 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:16,393 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:16,393 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:16,393 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:16,393 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:16,393 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:16,393 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:16,393 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:16,404 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:16,404 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:16,404 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:16,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:16,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:16,405 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:16,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:16,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:16,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:16,406 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:16,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:16,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:16,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:16,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:16,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:16,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,425 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:16,425 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,425 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:16,425 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:16,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:16,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:16,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:16,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:16,425 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:16,425 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:16,427 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:16,427 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:16,425 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:16,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:16,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:16,426 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:16,429 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:16,429 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:16,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:16,445 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=48.80 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/f2f5a0ad425f475dace0da5f5407d77a 2023-04-05 04:56:16,461 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/c13d76fee5d1496aa3ca21822f2f5a1b as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/c13d76fee5d1496aa3ca21822f2f5a1b 2023-04-05 04:56:16,474 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/c13d76fee5d1496aa3ca21822f2f5a1b, entries=250, sequenceid=50, filesize=16.2 K 2023-04-05 04:56:16,477 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/177f9cdd9dff4ad081e3409229f4ff64 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/177f9cdd9dff4ad081e3409229f4ff64 2023-04-05 04:56:16,477 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:16,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670636472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,490 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/177f9cdd9dff4ad081e3409229f4ff64, entries=150, sequenceid=50, filesize=11.6 K 2023-04-05 04:56:16,492 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/f2f5a0ad425f475dace0da5f5407d77a as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/f2f5a0ad425f475dace0da5f5407d77a 2023-04-05 04:56:16,511 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:16,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670636478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,516 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/f2f5a0ad425f475dace0da5f5407d77a, entries=150, sequenceid=50, filesize=11.6 K 2023-04-05 04:56:16,520 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~146.81 KB/150330, heapSize ~423.66 KB/433824, currentSize=33.54 KB/34350 for ed7bbafaddb97066dca4b03f35c9eee7 in 740ms, sequenceid=50, compaction requested=false 2023-04-05 04:56:16,523 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:16,527 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:16,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:16,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:16,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:16,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:16,567 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:16,567 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:16,570 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:16,571 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:16,571 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:16,571 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:16,573 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:16,573 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:16,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:16,573 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:16,573 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:16,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:16,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:16,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:16,574 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:16,581 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:16,581 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:16,582 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:16,582 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:16,583 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:16,586 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:16,586 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:16,587 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:16,587 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:16,587 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ed7bbafaddb97066dca4b03f35c9eee7 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-05 04:56:16,587 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=A 2023-04-05 04:56:16,588 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:16,588 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=B 2023-04-05 04:56:16,588 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:16,588 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=C 2023-04-05 04:56:16,588 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:16,686 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/c3b3f3f129ac4533bf397e3631b931d7 2023-04-05 04:56:16,795 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/5d110654783444028eba632aaff4444b 2023-04-05 04:56:16,889 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/3ad1ae4863a84c638f6607b8722cbceb 2023-04-05 04:56:16,891 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:16,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:16,902 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/c3b3f3f129ac4533bf397e3631b931d7 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/c3b3f3f129ac4533bf397e3631b931d7 2023-04-05 04:56:16,914 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/c3b3f3f129ac4533bf397e3631b931d7, entries=100, sequenceid=58, filesize=9.3 K 2023-04-05 04:56:16,915 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/5d110654783444028eba632aaff4444b as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/5d110654783444028eba632aaff4444b 2023-04-05 04:56:16,927 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/5d110654783444028eba632aaff4444b, entries=100, sequenceid=58, filesize=9.3 K 2023-04-05 04:56:16,931 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/3ad1ae4863a84c638f6607b8722cbceb as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/3ad1ae4863a84c638f6607b8722cbceb 2023-04-05 04:56:16,966 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/3ad1ae4863a84c638f6607b8722cbceb, entries=100, sequenceid=58, filesize=9.3 K 2023-04-05 04:56:16,969 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=73.80 KB/75570 for ed7bbafaddb97066dca4b03f35c9eee7 in 382ms, sequenceid=58, compaction requested=true 2023-04-05 04:56:16,969 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:16,969 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:16,969 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:16,971 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:16,971 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,971 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:16,971 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:16,976 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,976 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:16,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:16,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:16,976 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:16,976 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:16,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:16,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:16,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:16,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:16,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:16,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e55dc17[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:16,979 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:16,979 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:16,980 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:16,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:16,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:16,980 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:16,981 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:16,981 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:16,981 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,981 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:16,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:16,981 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ed7bbafaddb97066dca4b03f35c9eee7 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-05 04:56:16,982 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=A 2023-04-05 04:56:16,982 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:16,982 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=B 2023-04-05 04:56:16,982 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,982 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:16,982 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:16,982 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:16,982 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:16,982 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=C 2023-04-05 04:56:16,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,989 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:16,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:16,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:16,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:16,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:16,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:16,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:16,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:16,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:17,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:17,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:17,011 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:17,011 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:17,011 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:17,011 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:17,011 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:17,013 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,013 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:17,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:17,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,014 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:17,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:17,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:17,015 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:17,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:17,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:17,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:17,016 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:17,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:17,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,027 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,028 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:17,028 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:17,028 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,028 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:17,028 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:17,028 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:17,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:17,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:17,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:17,028 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:17,029 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:17,030 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:17,028 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,030 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:17,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:17,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:17,063 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:17,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670637047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,063 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:17,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670637050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:17,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49098 deadline: 1680670637050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,065 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:17,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670637050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:17,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49118 deadline: 1680670637052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,130 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:17,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:17,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:17,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:17,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:17,147 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:17,147 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:17,148 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:17,148 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:17,149 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,149 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:17,150 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,150 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,150 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:17,152 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:17,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:17,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:17,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:17,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:17,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:17,154 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:17,154 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:17,154 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:17,154 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:17,154 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:17,155 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:17,158 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:17,158 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:17,158 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:17,158 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:17,158 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:17,158 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:17,159 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:17,159 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:17,159 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,159 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:17,159 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:17,169 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:17,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670637165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,169 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:17,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:17,170 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:17,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670637165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,173 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:17,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49098 deadline: 1680670637166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,178 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:17,179 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,179 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:17,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:17,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670637167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:17,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:17,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:17,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:17,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6627acf7[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:17,182 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:17,182 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,183 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,184 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:17,184 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:17,184 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:17,186 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:17,186 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:17,186 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:17,187 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,188 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:17,188 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:17,188 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:17,188 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:17,188 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:17,188 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:17,188 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:17,190 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,190 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:17,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:17,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:17,191 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:17,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:17,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:17,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:17,192 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:17,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:17,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,194 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:17,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49118 deadline: 1680670637178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:17,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:17,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,204 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:17,204 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:17,204 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:17,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:17,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:17,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:17,204 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:17,204 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:17,205 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:17,206 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,206 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,206 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,206 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,206 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:17,206 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:17,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:17,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:17,248 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-05 04:56:17,307 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:17,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:17,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:17,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:17,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:17,329 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:17,329 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:17,330 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:17,330 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:17,330 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,331 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:17,335 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:17,335 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:17,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:17,335 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:17,336 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:17,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:17,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:17,337 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:17,337 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:17,337 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:17,337 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:17,337 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:17,338 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:17,339 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:17,339 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:17,339 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:17,339 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:17,339 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:17,339 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:17,339 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:17,339 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:17,339 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,339 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:17,339 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:17,343 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:17,343 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:17,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:17,344 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,344 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:17,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:17,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:17,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:17,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:17,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10156484[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:17,347 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:17,347 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,354 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,354 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,354 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:17,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,355 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:17,355 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:17,355 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:17,356 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,356 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:17,356 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:17,356 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:17,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:17,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:17,364 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-05 04:56:17,366 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-05 04:56:17,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:17,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:17,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:17,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:17,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:17,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:17,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:17,376 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:17,376 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:17,376 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:17,376 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:17,376 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:17,377 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,378 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:17,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,378 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:17,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:17,378 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:17,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:17,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:17,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:17,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:17,380 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,380 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:17,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:17,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670637375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:17,385 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,385 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:17,385 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:17,385 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:17,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:17,385 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:17,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:17,386 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:17,386 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,386 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,386 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,386 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:17,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:17,387 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:17,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670637380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:17,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:17,390 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:17,390 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:17,390 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:17,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49098 deadline: 1680670637377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670637382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,390 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:17,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:17,402 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:17,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49118 deadline: 1680670637398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,487 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=73 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/73a07b02fb1a49369267c6fdef4831fd 2023-04-05 04:56:17,491 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:17,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:17,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:17,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:17,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:17,518 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:17,518 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:17,523 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:17,523 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:17,525 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,525 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:17,526 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,526 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:17,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:17,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:17,527 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,527 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:17,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:17,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:17,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:17,529 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:17,538 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:17,538 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:17,538 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:17,538 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:17,548 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:17,550 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:17,550 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:17,550 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:17,550 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:17,550 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:17,550 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:17,551 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:17,551 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:17,551 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,557 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:17,557 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:17,558 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,558 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:17,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:17,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:17,559 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,559 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:17,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:17,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:17,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:17,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:17,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6799929e[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:17,561 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:17,561 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,562 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,562 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:17,562 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:17,562 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:17,562 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,562 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:17,563 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:17,563 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:17,563 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,563 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:17,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:17,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:17,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:17,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:17,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:17,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:17,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:17,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:17,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:17,570 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:17,570 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:17,570 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:17,570 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:17,570 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:17,584 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,584 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,584 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:17,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:17,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:17,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:17,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:17,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:17,588 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:17,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:17,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:17,611 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:17,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:17,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,614 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:17,615 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:17,615 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=73 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/73317ffc664844d39df8dfa8f641baf0 2023-04-05 04:56:17,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:17,615 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:17,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:17,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:17,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:17,615 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:17,615 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:17,616 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:17,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:17,616 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:17,616 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:17,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:17,616 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,617 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,617 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,617 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,690 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:17,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670637688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,692 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:17,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670637691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,703 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:17,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670637694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,704 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:17,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49098 deadline: 1680670637695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,707 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:17,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49118 deadline: 1680670637705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,716 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:17,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:17,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:17,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:17,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:17,724 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:17,724 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:17,725 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:17,725 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:17,726 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,726 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:17,727 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:17,727 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:17,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:17,728 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,728 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:17,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:17,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:17,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:17,729 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:17,730 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:17,730 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:17,730 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:17,732 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:17,734 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:17,734 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:17,734 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:17,734 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:17,734 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:17,734 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:17,735 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:17,735 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:17,735 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:17,735 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,735 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:17,735 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:17,745 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:17,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:17,746 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:17,746 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,746 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:17,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:17,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:17,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:17,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:17,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74c5f1e[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:17,748 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:17,748 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,751 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,752 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:17,752 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:17,752 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,752 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:17,752 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:17,753 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:17,753 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:17,753 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:17,758 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:17,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:17,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:17,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:17,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:17,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:17,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:17,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:17,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:17,763 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:17,763 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:17,763 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:17,763 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:17,763 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:17,768 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,768 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:17,768 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:17,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:17,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:17,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:17,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:17,770 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:17,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:17,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:17,772 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:17,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:17,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,779 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,779 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:17,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:17,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:17,779 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:17,780 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:17,780 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:17,779 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:17,783 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:17,779 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:17,783 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,783 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,783 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,783 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:17,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:17,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:17,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:17,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:17,883 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:17,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:17,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:17,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:17,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:17,890 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:17,890 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:17,894 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:17,894 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:17,895 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,895 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:17,896 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:17,896 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:17,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:17,897 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,897 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:17,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:17,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:17,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:17,898 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:17,898 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:17,898 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:17,898 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:17,898 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:17,900 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:17,901 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:17,901 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:17,901 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:17,901 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:17,901 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:17,901 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:17,901 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:17,901 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:17,901 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,901 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:17,901 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:17,905 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,905 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:17,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:17,906 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,906 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:17,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:17,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:17,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:17,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:17,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:17,908 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:17,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c571b14[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:17,908 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,918 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,918 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:17,918 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,918 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:17,918 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:17,918 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:17,920 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:17,920 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:17,920 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:17,924 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:17,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:17,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:17,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:17,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:17,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:17,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:17,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:17,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:17,927 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:17,927 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:17,927 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:17,927 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:17,927 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:17,935 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,935 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,935 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:17,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:17,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:17,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,935 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:17,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:17,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:17,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:17,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:17,938 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:17,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:17,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:17,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,943 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:17,943 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:17,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:17,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:17,943 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:17,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:17,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:17,944 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:17,944 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:17,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:17,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:17,946 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,946 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:17,946 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:17,946 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:17,946 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,946 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:17,946 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:17,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:18,044 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:18,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:18,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:18,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:18,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:18,054 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:18,054 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:18,055 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:18,055 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:18,056 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:18,056 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:18,059 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,059 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,059 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:18,066 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:18,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:18,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:18,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:18,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:18,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:18,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:18,068 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:18,070 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:18,070 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:18,070 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:18,070 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:18,073 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:18,073 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:18,073 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:18,073 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:18,073 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:18,073 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:18,073 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:18,074 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:18,074 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:18,074 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,075 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:18,075 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:18,076 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,076 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:18,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:18,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:18,076 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:18,076 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:18,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:18,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:18,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:18,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:18,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:18,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@675d755e[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:18,078 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:18,078 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:18,080 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:18,080 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,080 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:18,090 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=73 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/e21c5139e2ce48aa8df5b79215df8bec 2023-04-05 04:56:18,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:18,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:18,090 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:18,090 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:18,090 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:18,093 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,093 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:18,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:18,093 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:18,093 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:18,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:18,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:18,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:18,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:18,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:18,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:18,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:18,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:18,107 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:18,107 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:18,107 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:18,107 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:18,107 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:18,110 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/73a07b02fb1a49369267c6fdef4831fd as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/73a07b02fb1a49369267c6fdef4831fd 2023-04-05 04:56:18,110 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:18,110 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:18,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:18,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:18,111 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:18,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:18,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:18,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:18,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:18,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:18,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:18,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:18,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:18,113 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:18,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:18,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:18,115 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:18,118 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:18,118 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:18,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:18,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:18,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:18,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:18,121 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,121 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:18,121 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:18,121 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,121 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:18,121 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:18,121 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:18,122 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:18,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:18,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:18,124 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:18,124 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:18,125 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/73a07b02fb1a49369267c6fdef4831fd, entries=150, sequenceid=73, filesize=11.6 K 2023-04-05 04:56:18,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:18,131 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/73317ffc664844d39df8dfa8f641baf0 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/73317ffc664844d39df8dfa8f641baf0 2023-04-05 04:56:18,146 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/73317ffc664844d39df8dfa8f641baf0, entries=150, sequenceid=73, filesize=11.6 K 2023-04-05 04:56:18,150 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/e21c5139e2ce48aa8df5b79215df8bec as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/e21c5139e2ce48aa8df5b79215df8bec 2023-04-05 04:56:18,161 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/e21c5139e2ce48aa8df5b79215df8bec, entries=150, sequenceid=73, filesize=11.6 K 2023-04-05 04:56:18,163 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for ed7bbafaddb97066dca4b03f35c9eee7 in 1182ms, sequenceid=73, compaction requested=true 2023-04-05 04:56:18,163 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:18,166 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:56:18,166 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:56:18,166 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-05 04:56:18,166 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:56:18,170 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:56:18,175 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:56:18,177 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): ed7bbafaddb97066dca4b03f35c9eee7/B is initiating minor compaction (all files) 2023-04-05 04:56:18,177 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ed7bbafaddb97066dca4b03f35c9eee7/B in TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:18,177 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/046d7cbf21584d27ae1ec2fd07dfe460, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/177f9cdd9dff4ad081e3409229f4ff64, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/5d110654783444028eba632aaff4444b, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/73317ffc664844d39df8dfa8f641baf0] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp, totalSize=44.1 K 2023-04-05 04:56:18,178 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 046d7cbf21584d27ae1ec2fd07dfe460, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1680670574358 2023-04-05 04:56:18,179 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52220 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:56:18,180 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): ed7bbafaddb97066dca4b03f35c9eee7/A is initiating minor compaction (all files) 2023-04-05 04:56:18,181 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ed7bbafaddb97066dca4b03f35c9eee7/A in TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:18,181 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 177f9cdd9dff4ad081e3409229f4ff64, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680670574699 2023-04-05 04:56:18,181 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/430c0819709b4c71aa086159941e048e, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/c13d76fee5d1496aa3ca21822f2f5a1b, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/c3b3f3f129ac4533bf397e3631b931d7, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/73a07b02fb1a49369267c6fdef4831fd] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp, totalSize=51.0 K 2023-04-05 04:56:18,182 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 5d110654783444028eba632aaff4444b, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1680670575806 2023-04-05 04:56:18,182 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 430c0819709b4c71aa086159941e048e, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1680670574350 2023-04-05 04:56:18,184 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting c13d76fee5d1496aa3ca21822f2f5a1b, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680670574699 2023-04-05 04:56:18,185 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting c3b3f3f129ac4533bf397e3631b931d7, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1680670575806 2023-04-05 04:56:18,186 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 73317ffc664844d39df8dfa8f641baf0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=73, earliestPutTs=1680670576894 2023-04-05 04:56:18,188 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 73a07b02fb1a49369267c6fdef4831fd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=73, earliestPutTs=1680670576894 2023-04-05 04:56:18,208 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ed7bbafaddb97066dca4b03f35c9eee7 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-05 04:56:18,208 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=A 2023-04-05 04:56:18,208 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:18,208 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=B 2023-04-05 04:56:18,209 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:18,209 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=C 2023-04-05 04:56:18,209 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:18,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:18,224 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:18,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:18,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:18,234 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): ed7bbafaddb97066dca4b03f35c9eee7#B#compaction#13 average throughput is 0.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:56:18,235 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): ed7bbafaddb97066dca4b03f35c9eee7#A#compaction#14 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:56:18,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:18,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:18,236 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:18,236 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:18,239 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:18,239 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:18,240 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:18,240 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:18,243 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:18,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:18,243 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:18,244 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:18,244 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:18,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:18,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:18,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:18,245 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:18,251 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:18,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670638236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,254 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:18,254 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:18,254 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:18,254 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:18,263 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:18,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670638249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,264 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:18,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49098 deadline: 1680670638248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,264 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:18,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49118 deadline: 1680670638252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,265 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:18,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670638252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,270 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:18,270 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:18,271 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:18,271 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:18,271 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:18,271 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:18,271 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:18,271 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:18,271 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:18,271 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,271 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:18,271 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:18,282 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:18,282 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:18,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:18,285 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:18,285 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:18,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:18,285 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-05 04:56:18,286 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-05 04:56:18,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:18,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:18,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:18,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:18,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c7b0649[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:18,307 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:18,307 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:18,313 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/88ff7f71ed4045138801c3526c432bdd 2023-04-05 04:56:18,315 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:18,316 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,316 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:18,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:18,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:18,316 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:18,316 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:18,316 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:18,321 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,321 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:18,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:18,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:18,321 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:18,321 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:18,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:18,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:18,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:18,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:18,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:18,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:18,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:18,367 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:18,367 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:18,367 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:18,367 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:18,367 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:18,368 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:18,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670638365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,369 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:18,369 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:18,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:18,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:18,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:18,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:18,370 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:18,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:18,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670638365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,371 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:18,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:18,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:18,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:18,371 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:18,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49118 deadline: 1680670638367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:18,371 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:18,371 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49098 deadline: 1680670638367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:18,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:18,373 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/84550ac17ed549ae9ee1d88f8695f273 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/84550ac17ed549ae9ee1d88f8695f273 2023-04-05 04:56:18,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:18,373 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:18,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:18,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,384 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:18,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670638381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,385 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:18,385 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:18,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:18,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:18,385 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,386 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:18,388 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:18,388 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,388 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:18,388 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:18,388 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:18,388 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:18,388 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:18,389 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:18,389 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:18,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:18,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:18,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:18,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:18,403 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/edf70cf7007b419ca0e89cc06925b694 2023-04-05 04:56:18,405 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ed7bbafaddb97066dca4b03f35c9eee7/B of ed7bbafaddb97066dca4b03f35c9eee7 into 84550ac17ed549ae9ee1d88f8695f273(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:56:18,405 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:18,405 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7., storeName=ed7bbafaddb97066dca4b03f35c9eee7/B, priority=12, startTime=1680670578166; duration=0sec 2023-04-05 04:56:18,406 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:56:18,406 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:56:18,409 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:56:18,409 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): ed7bbafaddb97066dca4b03f35c9eee7/C is initiating minor compaction (all files) 2023-04-05 04:56:18,410 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ed7bbafaddb97066dca4b03f35c9eee7/C in TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:18,410 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/4c4cd0a96b4f46178a224a8ab1828bd8, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/f2f5a0ad425f475dace0da5f5407d77a, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/3ad1ae4863a84c638f6607b8722cbceb, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/e21c5139e2ce48aa8df5b79215df8bec] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp, totalSize=44.1 K 2023-04-05 04:56:18,411 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 4c4cd0a96b4f46178a224a8ab1828bd8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1680670574358 2023-04-05 04:56:18,413 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting f2f5a0ad425f475dace0da5f5407d77a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680670574699 2023-04-05 04:56:18,425 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 3ad1ae4863a84c638f6607b8722cbceb, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1680670575806 2023-04-05 04:56:18,429 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting e21c5139e2ce48aa8df5b79215df8bec, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=73, earliestPutTs=1680670576894 2023-04-05 04:56:18,440 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/4d52be4332e543e4a5e1a22eadeb1d32 2023-04-05 04:56:18,458 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/88ff7f71ed4045138801c3526c432bdd as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/88ff7f71ed4045138801c3526c432bdd 2023-04-05 04:56:18,466 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/88ff7f71ed4045138801c3526c432bdd, entries=250, sequenceid=92, filesize=16.2 K 2023-04-05 04:56:18,467 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/edf70cf7007b419ca0e89cc06925b694 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/edf70cf7007b419ca0e89cc06925b694 2023-04-05 04:56:18,475 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): ed7bbafaddb97066dca4b03f35c9eee7#C#compaction#17 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:56:18,487 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/edf70cf7007b419ca0e89cc06925b694, entries=150, sequenceid=92, filesize=11.6 K 2023-04-05 04:56:18,488 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/4d52be4332e543e4a5e1a22eadeb1d32 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/4d52be4332e543e4a5e1a22eadeb1d32 2023-04-05 04:56:18,488 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:18,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:18,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:18,513 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/4d52be4332e543e4a5e1a22eadeb1d32, entries=150, sequenceid=92, filesize=11.6 K 2023-04-05 04:56:18,516 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for ed7bbafaddb97066dca4b03f35c9eee7 in 308ms, sequenceid=92, compaction requested=false 2023-04-05 04:56:18,516 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:18,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:18,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:18,519 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:18,519 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:18,520 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:18,520 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:18,521 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:18,521 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:18,522 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,523 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/5b7a6109cea64ff29db470e06c2dfbcc as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/5b7a6109cea64ff29db470e06c2dfbcc 2023-04-05 04:56:18,523 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,523 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:18,524 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:18,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:18,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:18,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:18,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:18,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:18,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:18,526 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:18,526 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:18,527 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:18,527 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:18,528 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:18,529 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:18,529 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:18,530 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:18,538 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:18,538 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ed7bbafaddb97066dca4b03f35c9eee7 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-05 04:56:18,539 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=A 2023-04-05 04:56:18,539 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:18,539 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=B 2023-04-05 04:56:18,539 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:18,540 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=C 2023-04-05 04:56:18,540 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:18,551 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ed7bbafaddb97066dca4b03f35c9eee7/C of ed7bbafaddb97066dca4b03f35c9eee7 into 5b7a6109cea64ff29db470e06c2dfbcc(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:56:18,552 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:18,552 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7., storeName=ed7bbafaddb97066dca4b03f35c9eee7/C, priority=12, startTime=1680670578166; duration=0sec 2023-04-05 04:56:18,552 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:56:18,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:18,576 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:18,607 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:18,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49118 deadline: 1680670638598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,608 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:18,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49098 deadline: 1680670638602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,609 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:18,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670638605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,609 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:18,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670638605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,610 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:18,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670638607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,711 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:18,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49118 deadline: 1680670638709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,714 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:18,714 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:18,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49098 deadline: 1680670638710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670638710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,715 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:18,715 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:18,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670638711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670638711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,745 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/fb409db9b41845e38392a2ee1419572c as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/fb409db9b41845e38392a2ee1419572c 2023-04-05 04:56:18,753 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ed7bbafaddb97066dca4b03f35c9eee7/A of ed7bbafaddb97066dca4b03f35c9eee7 into fb409db9b41845e38392a2ee1419572c(size=11.7 K), total size for store is 27.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:56:18,754 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:18,754 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7., storeName=ed7bbafaddb97066dca4b03f35c9eee7/A, priority=12, startTime=1680670578165; duration=0sec 2023-04-05 04:56:18,754 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:56:18,915 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:18,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49118 deadline: 1680670638914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,919 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:18,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670638919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,924 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:18,924 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:18,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670638920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670638920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,924 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:18,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49098 deadline: 1680670638920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:18,970 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/e111ab7db17f4fedb8895d2162e9a8fc 2023-04-05 04:56:18,995 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/9daee39e174d47e98c61a84724a3dbf3 2023-04-05 04:56:19,221 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:19,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49118 deadline: 1680670639220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,224 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:19,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670639223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,228 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:19,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670639226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,228 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:19,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49098 deadline: 1680670639227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,230 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:19,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670639227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,424 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/0deb9640a91c48fe8f232923965cfca4 2023-04-05 04:56:19,433 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/e111ab7db17f4fedb8895d2162e9a8fc as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/e111ab7db17f4fedb8895d2162e9a8fc 2023-04-05 04:56:19,443 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/e111ab7db17f4fedb8895d2162e9a8fc, entries=100, sequenceid=106, filesize=9.3 K 2023-04-05 04:56:19,452 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/9daee39e174d47e98c61a84724a3dbf3 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/9daee39e174d47e98c61a84724a3dbf3 2023-04-05 04:56:19,462 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/9daee39e174d47e98c61a84724a3dbf3, entries=100, sequenceid=106, filesize=9.3 K 2023-04-05 04:56:19,466 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/0deb9640a91c48fe8f232923965cfca4 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/0deb9640a91c48fe8f232923965cfca4 2023-04-05 04:56:19,477 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/0deb9640a91c48fe8f232923965cfca4, entries=100, sequenceid=106, filesize=9.3 K 2023-04-05 04:56:19,479 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for ed7bbafaddb97066dca4b03f35c9eee7 in 941ms, sequenceid=106, compaction requested=true 2023-04-05 04:56:19,480 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:19,480 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:19,480 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:19,480 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:19,481 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,481 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:19,481 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:19,482 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:19,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:19,482 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:19,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:19,482 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:19,482 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:19,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:19,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:19,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:19,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:19,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@292a0477[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:19,484 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:19,484 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:19,485 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:19,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:19,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:19,485 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:19,485 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:19,485 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:19,485 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,485 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:19,486 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:19,486 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:19,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:19,486 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:19,486 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:19,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:19,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:19,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:19,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:19,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:19,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:19,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:19,493 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:19,493 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:19,493 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:19,493 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:19,493 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:19,495 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:19,496 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:19,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:19,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:19,496 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:19,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:19,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:19,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:19,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:19,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:19,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:19,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:19,497 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:19,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:19,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:19,500 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:19,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:19,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,510 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:19,510 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:19,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:19,510 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,510 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:19,510 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:19,510 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,510 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:19,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:19,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:19,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:19,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:19,510 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:19,510 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:19,510 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:19,511 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:19,511 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:19,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:19,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:19,612 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:19,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:19,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:19,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:19,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:19,617 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:19,617 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:19,618 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:19,618 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:19,619 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:19,619 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:19,620 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,620 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:19,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:19,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:19,620 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,620 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:19,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:19,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:19,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:19,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:19,621 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:19,625 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:19,625 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:19,625 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:19,625 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:19,626 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:19,626 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:19,626 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:19,626 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:19,626 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:19,627 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ed7bbafaddb97066dca4b03f35c9eee7 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-05 04:56:19,627 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=A 2023-04-05 04:56:19,627 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:19,627 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=B 2023-04-05 04:56:19,627 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:19,627 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=C 2023-04-05 04:56:19,627 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:19,654 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/b0225d37f88d4f708fc6f6a60a62c694 2023-04-05 04:56:19,696 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/74bc59a9bd5c4ca0a6dacacbf6b73c31 2023-04-05 04:56:19,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:19,729 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:19,751 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/e5d2fa9b0e9f4c6c98dd56941987eca6 2023-04-05 04:56:19,760 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/b0225d37f88d4f708fc6f6a60a62c694 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/b0225d37f88d4f708fc6f6a60a62c694 2023-04-05 04:56:19,761 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:19,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49118 deadline: 1680670639751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,761 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:19,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670639753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,762 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:19,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49098 deadline: 1680670639755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,764 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:19,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670639762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:19,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670639765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,773 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/b0225d37f88d4f708fc6f6a60a62c694, entries=150, sequenceid=127, filesize=11.6 K 2023-04-05 04:56:19,774 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/74bc59a9bd5c4ca0a6dacacbf6b73c31 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/74bc59a9bd5c4ca0a6dacacbf6b73c31 2023-04-05 04:56:19,789 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/74bc59a9bd5c4ca0a6dacacbf6b73c31, entries=150, sequenceid=127, filesize=11.6 K 2023-04-05 04:56:19,795 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/e5d2fa9b0e9f4c6c98dd56941987eca6 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/e5d2fa9b0e9f4c6c98dd56941987eca6 2023-04-05 04:56:19,804 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/e5d2fa9b0e9f4c6c98dd56941987eca6, entries=150, sequenceid=127, filesize=11.6 K 2023-04-05 04:56:19,806 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for ed7bbafaddb97066dca4b03f35c9eee7 in 180ms, sequenceid=127, compaction requested=true 2023-04-05 04:56:19,806 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:19,806 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:19,806 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:19,807 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:19,807 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,807 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:19,807 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:19,809 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,809 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:19,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:19,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:19,809 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:19,809 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:19,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:19,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:19,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:19,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:19,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:19,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dc135fd[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:19,811 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:19,811 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:19,812 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:19,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:19,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:19,812 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:19,812 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,812 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:19,812 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:19,812 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:19,813 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,813 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:19,813 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:19,813 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:19,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:19,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:19,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:19,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:19,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:19,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:19,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:19,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:19,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:19,823 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:19,823 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:19,823 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:19,823 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:19,823 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:19,826 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:19,826 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:19,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:19,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:19,826 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:19,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:19,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:19,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:19,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:19,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:19,827 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:19,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:19,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:19,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:19,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:19,828 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:19,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:19,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,837 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:19,837 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:19,837 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:19,837 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:19,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:19,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:19,838 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:19,838 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,838 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:19,838 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:19,838 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:19,838 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:19,838 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:19,838 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:19,838 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:19,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:19,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:19,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:19,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:19,866 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ed7bbafaddb97066dca4b03f35c9eee7 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-05 04:56:19,867 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=A 2023-04-05 04:56:19,867 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:19,867 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=B 2023-04-05 04:56:19,867 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:19,867 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=C 2023-04-05 04:56:19,867 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:19,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:19,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670639901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:19,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49098 deadline: 1680670639901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,911 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:19,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670639903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,911 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:19,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49118 deadline: 1680670639905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:19,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670639910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,938 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:19,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:19,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:19,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:19,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:19,945 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:19,945 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:19,946 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:19,946 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:19,946 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:19,946 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:19,948 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:19,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:19,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:19,948 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:19,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:19,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:19,949 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,949 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:19,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:19,949 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:19,950 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:19,950 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:19,950 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:19,950 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:19,951 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:19,951 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:19,951 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:19,951 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:19,951 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:19,952 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:19,952 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:19,952 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:19,952 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:19,952 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,952 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:19,952 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:19,960 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:19,960 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:19,960 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:19,960 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:19,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:19,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:19,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@413a4e9e[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:19,962 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:19,963 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:19,964 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:19,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:19,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:19,965 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:19,965 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:19,965 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:19,966 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,966 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:19,967 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:19,967 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:19,967 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:19,967 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:19,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:19,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:19,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:19,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:19,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:19,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:19,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:19,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:19,975 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:19,975 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:19,975 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:19,975 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:19,975 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:19,979 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:19,979 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:19,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:19,979 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:19,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:19,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:19,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:19,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:19,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:19,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:19,980 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:19,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:19,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:19,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:19,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:19,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:19,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:19,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:19,987 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,001 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:20,001 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:20,002 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:20,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:20,002 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:20,002 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:20,002 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,002 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,002 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,002 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,002 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:20,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:20,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:20,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:20,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:20,007 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:20,007 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:20,016 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:20,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49098 deadline: 1680670640012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,016 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:20,016 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:20,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670640012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49118 deadline: 1680670640013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,017 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:20,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670640013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,017 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:20,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670640015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,108 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:20,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:20,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:20,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:20,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:20,112 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:20,112 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:20,113 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:20,113 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:20,113 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,113 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:20,114 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,114 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:20,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:20,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:20,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:20,115 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,115 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:20,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:20,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:20,116 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:20,116 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:20,116 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:20,116 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:20,116 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:20,117 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:20,117 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:20,117 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:20,117 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:20,117 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:20,118 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:20,118 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:20,118 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:20,118 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:20,118 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,118 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:20,118 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:20,126 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,126 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:20,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:20,127 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,127 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:20,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:20,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:20,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:20,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:20,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:20,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@556741be[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:20,129 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:20,130 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,135 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,135 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:20,135 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:20,135 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:20,135 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,135 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:20,136 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:20,136 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:20,136 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:20,136 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:20,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:20,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:20,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:20,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:20,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:20,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:20,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:20,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:20,141 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:20,141 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:20,141 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:20,141 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:20,141 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:20,142 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,142 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:20,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,143 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:20,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:20,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:20,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:20,144 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:20,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:20,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:20,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:20,147 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:20,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:20,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,154 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:20,154 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:20,154 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:20,154 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:20,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:20,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:20,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:20,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:20,154 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:20,156 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:20,156 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:20,156 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:20,156 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,156 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:20,156 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:20,156 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,220 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:20,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670640218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,221 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:20,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49098 deadline: 1680670640218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,221 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:20,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670640219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,221 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:20,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49118 deadline: 1680670640219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,221 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:20,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670640219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,256 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:20,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:20,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:20,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:20,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:20,260 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:20,260 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:20,261 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:20,261 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:20,262 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,262 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:20,263 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:20,263 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:20,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:20,263 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,263 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:20,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:20,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:20,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:20,264 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:20,265 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:20,265 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:20,265 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:20,265 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:20,270 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:20,271 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:20,271 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:20,271 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:20,271 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:20,271 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:20,271 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:20,271 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:20,271 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:20,272 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,272 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:20,272 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:20,276 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,276 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:20,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:20,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:20,277 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,277 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:20,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:20,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:20,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:20,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:20,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c17fed9[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:20,279 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:20,279 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,280 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,280 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:20,280 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:20,280 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:20,281 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,281 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:20,281 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,281 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:20,281 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:20,281 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:20,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:20,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:20,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:20,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:20,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:20,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:20,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:20,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:20,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:20,285 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:20,285 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:20,285 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:20,285 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:20,285 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:20,293 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,293 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:20,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:20,293 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:20,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:20,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:20,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:20,294 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:20,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:20,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:20,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,295 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:20,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:20,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,303 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/c6488e8bc7fe41e58638bab84861f173 2023-04-05 04:56:20,304 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:20,304 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:20,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:20,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:20,304 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,304 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:20,304 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:20,304 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,304 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,304 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,304 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:20,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:20,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:20,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:20,305 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:20,306 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:20,306 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:20,343 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/c8b91a51144a4c289ef51a993b16f1bc 2023-04-05 04:56:20,406 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:20,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:20,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:20,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:20,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:20,410 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:20,410 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:20,411 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:20,411 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:20,412 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,412 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:20,412 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,412 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:20,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:20,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:20,413 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,413 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:20,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:20,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:20,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:20,413 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:20,414 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:20,414 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:20,414 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:20,414 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:20,415 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:20,415 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:20,415 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:20,415 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:20,415 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:20,415 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:20,415 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:20,415 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:20,415 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:20,415 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,415 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:20,415 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:20,417 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,417 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:20,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:20,417 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,417 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:20,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:20,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:20,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:20,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:20,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:20,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@683d487b[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:20,418 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:20,418 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,419 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,419 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:20,419 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:20,419 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,419 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:20,419 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:20,420 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,420 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:20,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:20,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:20,420 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:20,420 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:20,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:20,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:20,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:20,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:20,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:20,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:20,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:20,423 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:20,423 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:20,423 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:20,423 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:20,423 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:20,424 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,424 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:20,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:20,424 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:20,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,424 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:20,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:20,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:20,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:20,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:20,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:20,426 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:20,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:20,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,427 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:20,428 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:20,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:20,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:20,428 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,428 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:20,428 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:20,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:20,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:20,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:20,428 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:20,428 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:20,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:20,428 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,428 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,428 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,428 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:20,429 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:20,528 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:20,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670640522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,528 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:20,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49118 deadline: 1680670640522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,529 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:20,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:20,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670640524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670640524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,529 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:20,529 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:20,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49098 deadline: 1680670640525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:20,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:20,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:20,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:20,535 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:20,535 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:20,543 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:20,543 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:20,543 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,543 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:20,545 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,545 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:20,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:20,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:20,545 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,545 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:20,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:20,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:20,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:20,546 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:20,546 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:20,547 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:20,547 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:20,547 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:20,548 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:20,548 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:20,548 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:20,548 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:20,548 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:20,548 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:20,548 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:20,548 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:20,548 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:20,548 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,548 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:20,548 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:20,551 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,551 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:20,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:20,552 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:20,552 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:20,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:20,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:20,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:20,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:20,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7be0b55c[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:20,553 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:20,553 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,554 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,554 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:20,554 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:20,554 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:20,555 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,555 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:20,555 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:20,555 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,555 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:20,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:20,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:20,555 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:20,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:20,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:20,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:20,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:20,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:20,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:20,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:20,559 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:20,559 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:20,559 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:20,559 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:20,559 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:20,560 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,560 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:20,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:20,560 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:20,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:20,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:20,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:20,561 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:20,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:20,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:20,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,564 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:20,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:20,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,568 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,568 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:20,568 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:20,568 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,568 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:20,568 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:20,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:20,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:20,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:20,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:20,568 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,569 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,569 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:20,568 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:20,568 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:20,569 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:20,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:20,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:20,669 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:20,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:20,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:20,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:20,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:20,673 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:20,673 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:20,674 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:20,674 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:20,674 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,674 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:20,675 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:20,675 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:20,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:20,676 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:20,676 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:20,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:20,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:20,676 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:20,676 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:20,677 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:20,677 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:20,677 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:20,679 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:20,679 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:20,679 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:20,679 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:20,679 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:20,679 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:20,679 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:20,679 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:20,679 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:20,679 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,679 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:20,679 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:20,685 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,685 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:20,685 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,685 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:20,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:20,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:20,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:20,686 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:20,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14e5712f[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:20,686 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,688 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,688 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:20,688 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:20,688 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:20,689 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:20,689 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:20,689 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:20,690 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,690 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:20,690 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:20,690 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:20,690 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:20,690 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:20,690 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:20,690 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:20,692 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,692 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,692 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:20,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:20,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:20,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:20,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:20,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:20,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,698 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,699 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,704 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,704 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:20,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:20,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:20,704 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:20,704 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,704 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,704 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:20,704 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,704 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:20,704 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:20,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:20,704 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:20,704 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:20,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:20,706 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:20,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:20,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:20,763 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/7858562f20f44824a178b56731681416 2023-04-05 04:56:20,769 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/c6488e8bc7fe41e58638bab84861f173 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/c6488e8bc7fe41e58638bab84861f173 2023-04-05 04:56:20,776 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/c6488e8bc7fe41e58638bab84861f173, entries=200, sequenceid=142, filesize=14.1 K 2023-04-05 04:56:20,778 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/c8b91a51144a4c289ef51a993b16f1bc as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/c8b91a51144a4c289ef51a993b16f1bc 2023-04-05 04:56:20,787 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/c8b91a51144a4c289ef51a993b16f1bc, entries=150, sequenceid=142, filesize=11.8 K 2023-04-05 04:56:20,789 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/7858562f20f44824a178b56731681416 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/7858562f20f44824a178b56731681416 2023-04-05 04:56:20,798 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/7858562f20f44824a178b56731681416, entries=150, sequenceid=142, filesize=11.8 K 2023-04-05 04:56:20,800 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for ed7bbafaddb97066dca4b03f35c9eee7 in 934ms, sequenceid=142, compaction requested=true 2023-04-05 04:56:20,800 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:20,800 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:56:20,800 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 04:56:20,800 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 04:56:20,800 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:56:20,801 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:56:20,803 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57374 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 04:56:20,803 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64444 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 04:56:20,803 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): ed7bbafaddb97066dca4b03f35c9eee7/B is initiating minor compaction (all files) 2023-04-05 04:56:20,803 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): ed7bbafaddb97066dca4b03f35c9eee7/A is initiating minor compaction (all files) 2023-04-05 04:56:20,803 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ed7bbafaddb97066dca4b03f35c9eee7/B in TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:20,803 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ed7bbafaddb97066dca4b03f35c9eee7/A in TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:20,803 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/84550ac17ed549ae9ee1d88f8695f273, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/edf70cf7007b419ca0e89cc06925b694, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/9daee39e174d47e98c61a84724a3dbf3, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/74bc59a9bd5c4ca0a6dacacbf6b73c31, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/c8b91a51144a4c289ef51a993b16f1bc] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp, totalSize=56.0 K 2023-04-05 04:56:20,803 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/fb409db9b41845e38392a2ee1419572c, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/88ff7f71ed4045138801c3526c432bdd, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/e111ab7db17f4fedb8895d2162e9a8fc, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/b0225d37f88d4f708fc6f6a60a62c694, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/c6488e8bc7fe41e58638bab84861f173] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp, totalSize=62.9 K 2023-04-05 04:56:20,803 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 84550ac17ed549ae9ee1d88f8695f273, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=73, earliestPutTs=1680670576894 2023-04-05 04:56:20,804 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting fb409db9b41845e38392a2ee1419572c, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=73, earliestPutTs=1680670576894 2023-04-05 04:56:20,806 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting edf70cf7007b419ca0e89cc06925b694, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1680670577024 2023-04-05 04:56:20,806 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 88ff7f71ed4045138801c3526c432bdd, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1680670577024 2023-04-05 04:56:20,806 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 9daee39e174d47e98c61a84724a3dbf3, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1680670578248 2023-04-05 04:56:20,807 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:20,807 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting e111ab7db17f4fedb8895d2162e9a8fc, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1680670578248 2023-04-05 04:56:20,807 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 74bc59a9bd5c4ca0a6dacacbf6b73c31, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1680670578583 2023-04-05 04:56:20,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:20,808 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting c8b91a51144a4c289ef51a993b16f1bc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1680670579748 2023-04-05 04:56:20,808 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting b0225d37f88d4f708fc6f6a60a62c694, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1680670578583 2023-04-05 04:56:20,808 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting c6488e8bc7fe41e58638bab84861f173, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1680670579748 2023-04-05 04:56:20,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:20,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:20,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:20,814 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:20,814 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:20,814 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:20,814 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:20,815 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,815 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:20,816 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:20,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:20,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:20,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:20,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:20,818 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:20,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:20,818 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:20,819 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:20,819 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:20,830 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:20,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:20,830 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:20,831 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:20,831 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:20,837 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): ed7bbafaddb97066dca4b03f35c9eee7#A#compaction#27 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:56:20,839 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:20,839 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:20,839 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:20,839 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:20,839 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ed7bbafaddb97066dca4b03f35c9eee7 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-05 04:56:20,840 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=A 2023-04-05 04:56:20,840 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:20,840 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=B 2023-04-05 04:56:20,840 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:20,840 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=C 2023-04-05 04:56:20,840 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:20,847 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): ed7bbafaddb97066dca4b03f35c9eee7#B#compaction#28 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:56:20,905 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/1a9555663ff8406c8c5a17b5cc0a430f as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/1a9555663ff8406c8c5a17b5cc0a430f 2023-04-05 04:56:20,905 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=159 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/681e655b6f1b42c5a83d07ea1fc933d9 2023-04-05 04:56:20,915 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ed7bbafaddb97066dca4b03f35c9eee7/B of ed7bbafaddb97066dca4b03f35c9eee7 into 1a9555663ff8406c8c5a17b5cc0a430f(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:56:20,916 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:20,916 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7., storeName=ed7bbafaddb97066dca4b03f35c9eee7/B, priority=11, startTime=1680670580800; duration=0sec 2023-04-05 04:56:20,917 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:56:20,917 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 04:56:20,921 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57374 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 04:56:20,921 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): ed7bbafaddb97066dca4b03f35c9eee7/C is initiating minor compaction (all files) 2023-04-05 04:56:20,921 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ed7bbafaddb97066dca4b03f35c9eee7/C in TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:20,921 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/5b7a6109cea64ff29db470e06c2dfbcc, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/4d52be4332e543e4a5e1a22eadeb1d32, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/0deb9640a91c48fe8f232923965cfca4, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/e5d2fa9b0e9f4c6c98dd56941987eca6, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/7858562f20f44824a178b56731681416] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp, totalSize=56.0 K 2023-04-05 04:56:20,922 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 5b7a6109cea64ff29db470e06c2dfbcc, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=73, earliestPutTs=1680670576894 2023-04-05 04:56:20,923 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 4d52be4332e543e4a5e1a22eadeb1d32, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1680670577024 2023-04-05 04:56:20,924 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 0deb9640a91c48fe8f232923965cfca4, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1680670578248 2023-04-05 04:56:20,924 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting e5d2fa9b0e9f4c6c98dd56941987eca6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1680670578583 2023-04-05 04:56:20,925 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 7858562f20f44824a178b56731681416, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1680670579748 2023-04-05 04:56:20,951 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): ed7bbafaddb97066dca4b03f35c9eee7#C#compaction#31 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:56:20,958 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=159 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/78005fe6505d4c23b7b2c5cee317cbea 2023-04-05 04:56:20,988 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=159 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/9db0f55c83f941109c2f393aa1bbb904 2023-04-05 04:56:20,999 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/681e655b6f1b42c5a83d07ea1fc933d9 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/681e655b6f1b42c5a83d07ea1fc933d9 2023-04-05 04:56:21,008 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/681e655b6f1b42c5a83d07ea1fc933d9, entries=150, sequenceid=159, filesize=11.8 K 2023-04-05 04:56:21,010 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/78005fe6505d4c23b7b2c5cee317cbea as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/78005fe6505d4c23b7b2c5cee317cbea 2023-04-05 04:56:21,018 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/78005fe6505d4c23b7b2c5cee317cbea, entries=150, sequenceid=159, filesize=11.8 K 2023-04-05 04:56:21,023 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/9db0f55c83f941109c2f393aa1bbb904 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/9db0f55c83f941109c2f393aa1bbb904 2023-04-05 04:56:21,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:21,033 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:21,047 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/9db0f55c83f941109c2f393aa1bbb904, entries=150, sequenceid=159, filesize=11.8 K 2023-04-05 04:56:21,099 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=13.42 KB/13740 for ed7bbafaddb97066dca4b03f35c9eee7 in 260ms, sequenceid=159, compaction requested=false 2023-04-05 04:56:21,099 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:21,099 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:21,099 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:21,100 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:21,100 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,100 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:21,100 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:21,102 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,102 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:21,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:21,102 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,102 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:21,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:21,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:21,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:21,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:21,104 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ed7bbafaddb97066dca4b03f35c9eee7 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 04:56:21,104 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=A 2023-04-05 04:56:21,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:21,104 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:21,104 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:21,104 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,104 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=B 2023-04-05 04:56:21,105 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:21,105 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=C 2023-04-05 04:56:21,105 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:21,106 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@797b45c[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:21,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,106 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:21,106 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,106 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:21,106 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:21,106 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:21,107 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:21,107 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,107 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:21,108 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:21,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:21,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:21,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:21,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:21,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:21,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:21,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:21,113 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:21,113 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:21,113 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:21,113 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:21,113 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:21,120 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,121 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:21,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,121 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:21,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:21,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:21,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:21,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,122 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:21,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:21,124 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:21,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,130 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:21,130 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:21,130 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,130 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:21,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:21,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:21,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:21,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:21,130 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:21,130 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:21,131 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:21,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:21,131 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:21,131 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:21,131 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,131 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,131 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:21,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680670581132, max timestamp=1680670581147Num uniques 859; 2023-04-05 04:56:21,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-05 04:56:21,163 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:21,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670641155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,164 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:21,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49118 deadline: 1680670641157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,165 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:21,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670641157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,165 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:21,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670641159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,170 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:21,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49098 deadline: 1680670641164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,231 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:21,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:21,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:21,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:21,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:21,236 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:21,236 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:21,236 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:21,236 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:21,236 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,236 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:21,237 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:21,237 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:21,237 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:21,237 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:21,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:21,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:21,238 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:21,238 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:21,238 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:21,238 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:21,238 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:21,238 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:21,238 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:21,238 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:21,238 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:21,238 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:21,239 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:21,239 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:21,239 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:21,239 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:21,239 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,239 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:21,239 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:21,240 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,240 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:21,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:21,240 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,240 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:21,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:21,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:21,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:21,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:21,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56b5a45[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:21,241 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:21,241 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,242 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,242 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:21,242 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:21,242 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:21,242 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,242 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:21,243 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,243 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:21,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:21,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:21,243 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:21,243 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:21,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:21,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:21,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:21,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:21,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:21,245 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:21,245 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:21,245 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:21,245 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:21,245 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:21,246 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,246 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,246 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:21,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:21,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:21,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:21,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:21,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:21,246 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:21,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,248 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:21,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,249 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:21,249 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:21,249 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:21,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:21,249 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:21,249 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,250 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,250 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,251 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,251 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:21,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:21,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:21,251 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:21,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:21,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:21,251 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:21,251 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:21,267 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:21,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670641265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,267 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:21,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49118 deadline: 1680670641266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:21,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670641266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:21,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670641266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,275 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:21,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49098 deadline: 1680670641274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,292 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/0fca916428ce4e7dae6f208ac9f432f2 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/0fca916428ce4e7dae6f208ac9f432f2 2023-04-05 04:56:21,300 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ed7bbafaddb97066dca4b03f35c9eee7/A of ed7bbafaddb97066dca4b03f35c9eee7 into 0fca916428ce4e7dae6f208ac9f432f2(size=12.1 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:56:21,300 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:21,300 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7., storeName=ed7bbafaddb97066dca4b03f35c9eee7/A, priority=11, startTime=1680670580800; duration=0sec 2023-04-05 04:56:21,301 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:56:21,351 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:21,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:21,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:21,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:21,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:21,356 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:21,356 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:21,356 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:21,356 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:21,357 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,357 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:21,357 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:21,357 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:21,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:21,358 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,358 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:21,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:21,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:21,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:21,358 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:21,359 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:21,359 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:21,359 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:21,359 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:21,359 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:21,359 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:21,359 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:21,359 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:21,359 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:21,359 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:21,359 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:21,359 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:21,359 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:21,359 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,359 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:21,359 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:21,360 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,360 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:21,361 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,361 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:21,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:21,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:21,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bdf55e1[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:21,363 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:21,363 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,364 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,364 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:21,364 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:21,364 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:21,364 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,364 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:21,365 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:21,365 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,365 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:21,365 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:21,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:21,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:21,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:21,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:21,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:21,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:21,368 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:21,368 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:21,368 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:21,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:21,368 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:21,368 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:21,369 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,369 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,369 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:21,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:21,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:21,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:21,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:21,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:21,370 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:21,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,371 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:21,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,374 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,374 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:21,374 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:21,374 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,374 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,374 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:21,374 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:21,374 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,374 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:21,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:21,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:21,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:21,374 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:21,374 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:21,375 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:21,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:21,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:21,379 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/1a4f37425f7a462f844cdd969022a343 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/1a4f37425f7a462f844cdd969022a343 2023-04-05 04:56:21,387 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ed7bbafaddb97066dca4b03f35c9eee7/C of ed7bbafaddb97066dca4b03f35c9eee7 into 1a4f37425f7a462f844cdd969022a343(size=12.1 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:56:21,387 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:21,387 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7., storeName=ed7bbafaddb97066dca4b03f35c9eee7/C, priority=11, startTime=1680670580801; duration=0sec 2023-04-05 04:56:21,387 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:56:21,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:21,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670641469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:21,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49118 deadline: 1680670641470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:21,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670641471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,474 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:21,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670641471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,475 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:21,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:21,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:21,479 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:21,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49098 deadline: 1680670641478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:21,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:21,479 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:21,480 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:21,480 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:21,480 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:21,480 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,480 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:21,481 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,481 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:21,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:21,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:21,481 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,481 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:21,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:21,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:21,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:21,482 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:21,483 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:21,483 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:21,483 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:21,483 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:21,483 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:21,483 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:21,483 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:21,483 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:21,483 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:21,483 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:21,483 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:21,483 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:21,483 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:21,483 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,483 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:21,483 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:21,484 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,484 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:21,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:21,485 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,485 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:21,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:21,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:21,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:21,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:21,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@440c3fbc[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:21,486 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:21,486 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,487 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,487 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:21,487 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:21,487 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,487 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:21,487 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:21,488 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:21,488 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:21,488 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,488 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:21,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:21,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:21,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:21,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:21,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:21,490 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:21,490 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:21,490 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:21,490 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:21,490 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:21,491 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,491 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:21,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,491 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:21,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:21,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:21,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:21,492 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:21,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:21,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,493 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:21,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,504 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:21,504 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:21,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:21,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:21,504 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:21,504 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:21,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:21,504 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:21,505 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:21,505 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:21,505 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,505 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,505 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,505 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,505 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:21,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:21,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:21,522 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/1bbc70f0ac1542c8b14a2ba03cd702f3 2023-04-05 04:56:21,605 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:21,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:21,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:21,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:21,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:21,609 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:21,609 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:21,609 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:21,609 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:21,609 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,609 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:21,610 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,610 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:21,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:21,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:21,610 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,610 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:21,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:21,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:21,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:21,611 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:21,611 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:21,611 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:21,611 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:21,611 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:21,611 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:21,611 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:21,611 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:21,611 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:21,612 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:21,612 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:21,612 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:21,612 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:21,612 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:21,612 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,612 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:21,612 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:21,613 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,613 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:21,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:21,613 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,613 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:21,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:21,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:21,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:21,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:21,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e834635[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:21,618 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:21,618 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,629 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,629 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:21,629 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:21,629 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,629 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:21,629 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:21,635 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,635 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:21,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:21,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:21,635 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:21,635 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:21,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:21,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:21,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:21,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:21,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:21,637 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:21,637 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:21,637 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:21,637 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:21,637 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:21,638 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,638 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:21,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,639 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:21,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:21,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:21,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:21,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:21,639 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:21,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,641 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:21,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,643 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,643 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:21,643 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:21,643 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,643 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,643 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,643 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,643 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:21,643 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:21,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:21,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:21,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:21,643 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:21,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:21,643 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:21,644 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:21,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:21,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:21,744 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:21,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:21,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:21,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:21,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:21,748 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:21,748 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:21,748 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:21,748 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:21,748 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,748 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:21,749 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:21,749 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:21,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:21,750 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,750 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:21,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:21,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:21,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:21,750 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:21,750 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:21,751 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:21,751 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:21,751 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:21,751 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:21,751 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:21,751 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:21,751 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:21,751 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:21,751 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:21,751 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:21,752 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:21,752 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:21,752 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,752 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:21,752 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:21,753 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,753 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:21,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:21,753 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,753 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:21,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:21,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:21,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:21,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:21,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21e582d2[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:21,755 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:21,755 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,755 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,756 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:21,756 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:21,756 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:21,756 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,756 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:21,756 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,756 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:21,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:21,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:21,756 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:21,757 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:21,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:21,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:21,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:21,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:21,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:21,758 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:21,758 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:21,758 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:21,758 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:21,758 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:21,760 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,760 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:21,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,760 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:21,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:21,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:21,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:21,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:21,761 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:21,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,764 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:21,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,770 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,770 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:21,771 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:21,771 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:21,771 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:21,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:21,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:21,771 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,771 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,771 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,771 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:21,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:21,771 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:21,771 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:21,771 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:21,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:21,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:21,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:21,775 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:21,775 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:21,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49118 deadline: 1680670641774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670641774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,776 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:21,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670641775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,780 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:21,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670641777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,782 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:21,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49098 deadline: 1680670641780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,871 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:21,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:21,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:21,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:21,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:21,876 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:21,876 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:21,876 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:21,876 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:21,877 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,877 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:21,877 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:21,877 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:21,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:21,878 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,878 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:21,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:21,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:21,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:21,878 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:21,879 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:21,879 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:21,879 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:21,879 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:21,879 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:21,879 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:21,879 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:21,879 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:21,879 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:21,879 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:21,879 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:21,879 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:21,879 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:21,879 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,880 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:21,880 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:21,880 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,881 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:21,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:21,881 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:21,881 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:21,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:21,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:21,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:21,882 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:21,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2014bb52[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:21,882 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,883 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,883 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:21,883 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:21,883 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:21,883 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,883 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:21,884 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:21,884 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,884 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:21,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:21,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:21,884 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:21,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:21,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:21,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:21,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:21,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:21,886 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:21,886 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:21,886 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:21,886 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:21,886 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:21,887 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,887 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,888 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:21,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:21,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:21,888 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:21,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:21,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,889 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:21,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:21,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,891 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,891 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:21,891 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:21,891 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:21,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:21,891 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:21,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:21,892 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:21,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:21,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:21,892 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,892 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:21,892 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,892 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:21,892 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:21,892 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:21,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:21,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:21,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:21,938 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/a5032e3edd9746f29de2e6f00dba272f 2023-04-05 04:56:21,954 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/381717362efc439fae4b2fa7bbce368e 2023-04-05 04:56:21,961 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/1bbc70f0ac1542c8b14a2ba03cd702f3 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/1bbc70f0ac1542c8b14a2ba03cd702f3 2023-04-05 04:56:21,967 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/1bbc70f0ac1542c8b14a2ba03cd702f3, entries=150, sequenceid=170, filesize=11.8 K 2023-04-05 04:56:21,969 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/a5032e3edd9746f29de2e6f00dba272f as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/a5032e3edd9746f29de2e6f00dba272f 2023-04-05 04:56:21,984 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/a5032e3edd9746f29de2e6f00dba272f, entries=150, sequenceid=170, filesize=11.8 K 2023-04-05 04:56:21,988 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/381717362efc439fae4b2fa7bbce368e as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/381717362efc439fae4b2fa7bbce368e 2023-04-05 04:56:21,992 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:21,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:21,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:21,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:21,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:21,999 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:21,999 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:21,999 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:21,999 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:21,999 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:21,999 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:22,001 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/381717362efc439fae4b2fa7bbce368e, entries=150, sequenceid=170, filesize=11.8 K 2023-04-05 04:56:22,001 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:22,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:22,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:22,002 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:22,002 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,003 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:22,003 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=131.94 KB/135110 for ed7bbafaddb97066dca4b03f35c9eee7 in 898ms, sequenceid=170, compaction requested=true 2023-04-05 04:56:22,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:22,003 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:22,003 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:56:22,003 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:56:22,003 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:56:22,003 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:56:22,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:22,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:22,003 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:56:22,003 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:22,004 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:56:22,004 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:56:22,004 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): ed7bbafaddb97066dca4b03f35c9eee7/B is initiating minor compaction (all files) 2023-04-05 04:56:22,004 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): ed7bbafaddb97066dca4b03f35c9eee7/A is initiating minor compaction (all files) 2023-04-05 04:56:22,004 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ed7bbafaddb97066dca4b03f35c9eee7/B in TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:22,004 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ed7bbafaddb97066dca4b03f35c9eee7/A in TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:22,005 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/0fca916428ce4e7dae6f208ac9f432f2, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/681e655b6f1b42c5a83d07ea1fc933d9, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/1bbc70f0ac1542c8b14a2ba03cd702f3] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp, totalSize=35.6 K 2023-04-05 04:56:22,005 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/1a9555663ff8406c8c5a17b5cc0a430f, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/78005fe6505d4c23b7b2c5cee317cbea, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/a5032e3edd9746f29de2e6f00dba272f] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp, totalSize=35.6 K 2023-04-05 04:56:22,005 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 0fca916428ce4e7dae6f208ac9f432f2, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1680670579748 2023-04-05 04:56:22,007 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:22,005 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 1a9555663ff8406c8c5a17b5cc0a430f, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1680670579748 2023-04-05 04:56:22,007 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:22,008 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:22,008 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:22,008 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:22,008 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:22,008 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:22,008 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:22,008 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:22,008 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 681e655b6f1b42c5a83d07ea1fc933d9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=159, earliestPutTs=1680670579900 2023-04-05 04:56:22,008 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ed7bbafaddb97066dca4b03f35c9eee7 3/3 column families, dataSize=131.94 KB heapSize=382.03 KB 2023-04-05 04:56:22,008 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 78005fe6505d4c23b7b2c5cee317cbea, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=159, earliestPutTs=1680670579900 2023-04-05 04:56:22,008 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=A 2023-04-05 04:56:22,009 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:56:22,009 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=B 2023-04-05 04:56:22,009 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 1bbc70f0ac1542c8b14a2ba03cd702f3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1680670581043 2023-04-05 04:56:22,009 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting a5032e3edd9746f29de2e6f00dba272f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1680670581043 2023-04-05 04:56:22,009 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:56:22,009 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=C 2023-04-05 04:56:22,009 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:56:22,026 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/6cb305b077864af48242ff2c437b7121 2023-04-05 04:56:22,027 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): ed7bbafaddb97066dca4b03f35c9eee7#B#compaction#37 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:56:22,037 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): ed7bbafaddb97066dca4b03f35c9eee7#A#compaction#38 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:56:22,061 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/bc8a097904dc44dab24e1828eb2a9dc7 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/bc8a097904dc44dab24e1828eb2a9dc7 2023-04-05 04:56:22,070 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ed7bbafaddb97066dca4b03f35c9eee7/A of ed7bbafaddb97066dca4b03f35c9eee7 into bc8a097904dc44dab24e1828eb2a9dc7(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:56:22,070 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:22,070 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7., storeName=ed7bbafaddb97066dca4b03f35c9eee7/A, priority=13, startTime=1680670582003; duration=0sec 2023-04-05 04:56:22,070 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:56:22,070 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:56:22,072 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:56:22,072 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): ed7bbafaddb97066dca4b03f35c9eee7/C is initiating minor compaction (all files) 2023-04-05 04:56:22,072 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ed7bbafaddb97066dca4b03f35c9eee7/C in TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:22,072 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/1a4f37425f7a462f844cdd969022a343, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/9db0f55c83f941109c2f393aa1bbb904, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/381717362efc439fae4b2fa7bbce368e] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp, totalSize=35.6 K 2023-04-05 04:56:22,073 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 1a4f37425f7a462f844cdd969022a343, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1680670579748 2023-04-05 04:56:22,073 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 9db0f55c83f941109c2f393aa1bbb904, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=159, earliestPutTs=1680670579900 2023-04-05 04:56:22,074 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 381717362efc439fae4b2fa7bbce368e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1680670581043 2023-04-05 04:56:22,081 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/f75054eb216948b8a5812b01ae9a11f7 2023-04-05 04:56:22,089 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): ed7bbafaddb97066dca4b03f35c9eee7#C#compaction#40 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:56:22,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:22,280 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:22,289 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:22,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49118 deadline: 1680670642285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,289 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:22,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670642286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,289 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:22,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670642287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,290 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:22,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670642287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,290 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:22,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49098 deadline: 1680670642289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,391 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:22,393 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:22,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670642390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,394 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:22,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49118 deadline: 1680670642391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,396 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:22,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670642392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,397 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:22,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49098 deadline: 1680670642392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670642390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,440 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/1df758f917c64c86824b85e2284178de as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/1df758f917c64c86824b85e2284178de 2023-04-05 04:56:22,449 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ed7bbafaddb97066dca4b03f35c9eee7/B of ed7bbafaddb97066dca4b03f35c9eee7 into 1df758f917c64c86824b85e2284178de(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:56:22,449 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:22,449 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7., storeName=ed7bbafaddb97066dca4b03f35c9eee7/B, priority=13, startTime=1680670582003; duration=0sec 2023-04-05 04:56:22,449 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:56:22,505 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/798aff70a68e4034aad6d62b194ec086 2023-04-05 04:56:22,510 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/9ae8be7063fd4972a5def3b473d7f4f2 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/9ae8be7063fd4972a5def3b473d7f4f2 2023-04-05 04:56:22,514 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/6cb305b077864af48242ff2c437b7121 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/6cb305b077864af48242ff2c437b7121 2023-04-05 04:56:22,518 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ed7bbafaddb97066dca4b03f35c9eee7/C of ed7bbafaddb97066dca4b03f35c9eee7 into 9ae8be7063fd4972a5def3b473d7f4f2(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:56:22,518 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:22,518 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7., storeName=ed7bbafaddb97066dca4b03f35c9eee7/C, priority=13, startTime=1680670582003; duration=0sec 2023-04-05 04:56:22,518 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:56:22,520 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/6cb305b077864af48242ff2c437b7121, entries=150, sequenceid=198, filesize=11.8 K 2023-04-05 04:56:22,520 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.StoreScanner(953): StoreScanner already has the close lock. There is no need to updateReaders 2023-04-05 04:56:22,521 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/f75054eb216948b8a5812b01ae9a11f7 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/f75054eb216948b8a5812b01ae9a11f7 2023-04-05 04:56:22,526 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/f75054eb216948b8a5812b01ae9a11f7, entries=150, sequenceid=198, filesize=11.8 K 2023-04-05 04:56:22,527 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/798aff70a68e4034aad6d62b194ec086 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/798aff70a68e4034aad6d62b194ec086 2023-04-05 04:56:22,532 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/798aff70a68e4034aad6d62b194ec086, entries=150, sequenceid=198, filesize=11.8 K 2023-04-05 04:56:22,533 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~131.94 KB/135110, heapSize ~381.98 KB/391152, currentSize=46.96 KB/48090 for ed7bbafaddb97066dca4b03f35c9eee7 in 524ms, sequenceid=198, compaction requested=false 2023-04-05 04:56:22,533 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:22,533 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:22,533 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:22,533 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:22,533 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,533 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:22,533 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:22,534 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:22,534 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:22,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:22,534 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:22,534 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:22,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:22,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5409133[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:22,536 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:22,536 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:22,536 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:22,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:22,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:22,537 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:22,537 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:22,537 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:22,537 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,537 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:22,537 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:22,537 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,537 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:22,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:22,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:22,537 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:22,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:22,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:22,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:22,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:22,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:22,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:22,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:22,539 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:22,539 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:22,539 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:22,540 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:22,540 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:22,541 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:22,541 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:22,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:22,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:22,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:22,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:22,541 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:22,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:22,541 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:22,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:22,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:22,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:22,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:22,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:22,542 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:22,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:22,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:22,544 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,544 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:22,544 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:22,544 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,544 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:22,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,544 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:22,544 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:22,544 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:22,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:22,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:22,544 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:22,545 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:22,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:22,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:22,545 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:22,545 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:22,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:22,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:22,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:22,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:22,598 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ed7bbafaddb97066dca4b03f35c9eee7 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-05 04:56:22,598 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=A 2023-04-05 04:56:22,598 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:22,598 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=B 2023-04-05 04:56:22,599 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:22,599 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=C 2023-04-05 04:56:22,599 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:22,617 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:22,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49098 deadline: 1680670642614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,617 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:22,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670642615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,617 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:22,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670642616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,617 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:22,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49118 deadline: 1680670642616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:22,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670642617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,645 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:22,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:22,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:22,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:22,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:22,649 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:22,649 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:22,649 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:22,649 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:22,649 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:22,649 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:22,650 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:22,650 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:22,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:22,650 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,650 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:22,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:22,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:22,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:22,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:22,651 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:22,651 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:22,651 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:22,651 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:22,651 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:22,651 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:22,651 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:22,651 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:22,651 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:22,651 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:22,651 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:22,651 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:22,652 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:22,652 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:22,652 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,652 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:22,652 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:22,652 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,652 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:22,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:22,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:22,653 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:22,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:22,653 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:22,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:22,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:22,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:22,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:22,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6196dd40[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:22,654 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:22,654 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:22,654 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:22,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:22,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:22,655 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:22,655 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:22,655 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,655 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:22,655 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:22,656 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,656 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:22,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:22,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:22,656 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:22,656 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:22,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:22,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:22,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:22,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:22,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:22,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:22,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:22,659 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:22,659 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:22,659 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:22,659 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:22,659 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:22,660 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:22,660 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:22,660 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:22,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:22,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:22,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:22,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:22,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:22,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:22,661 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:22,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:22,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:22,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:22,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:22,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:22,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,662 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:22,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:22,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,664 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:22,664 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:22,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:22,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:22,664 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:22,664 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,664 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:22,664 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:22,664 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,664 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:22,664 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:22,664 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:22,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:22,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:22,665 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:22,665 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:22,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:22,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:22,666 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:22,720 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:22,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49098 deadline: 1680670642718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,720 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:22,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670642719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,721 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:22,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670642719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,721 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:22,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49118 deadline: 1680670642720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,723 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:22,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670642722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,765 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:22,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:22,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:22,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:22,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:22,768 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:22,768 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:22,768 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:22,768 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:22,768 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:22,768 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:22,769 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:22,769 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:22,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:22,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:22,769 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,770 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:22,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:22,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:22,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:22,770 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:22,770 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:22,770 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:22,770 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:22,770 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:22,770 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:22,770 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:22,770 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:22,770 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:22,771 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:22,771 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:22,771 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:22,771 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:22,771 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:22,771 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,771 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:22,771 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:22,772 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,772 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:22,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:22,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:22,772 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:22,772 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:22,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:22,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:22,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:22,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:22,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:22,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ce820b2[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:22,773 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:22,773 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:22,774 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:22,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:22,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:22,774 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,774 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:22,774 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:22,774 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:22,774 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:22,775 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,775 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:22,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:22,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:22,775 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:22,775 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:22,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:22,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:22,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:22,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:22,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:22,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:22,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:22,777 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:22,777 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:22,777 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:22,777 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:22,777 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:22,778 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:22,778 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:22,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:22,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:22,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:22,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:22,778 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:22,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:22,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:22,779 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:22,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:22,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:22,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:22,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:22,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:22,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,780 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:22,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:22,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,781 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,781 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:22,781 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:22,781 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:22,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:22,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:22,781 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:22,781 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:22,782 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,782 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:22,782 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:22,782 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:22,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:22,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:22,782 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:22,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:22,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:22,782 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:22,882 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:22,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:22,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:22,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:22,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:22,886 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:22,886 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:22,886 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:22,886 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:22,887 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:22,887 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:22,887 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,887 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:22,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:22,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:22,887 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,887 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:22,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:22,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:22,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:22,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:22,888 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:22,889 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:22,889 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:22,889 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:22,889 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:22,889 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:22,889 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:22,889 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:22,889 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:22,889 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:22,889 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:22,889 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:22,889 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:22,889 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:22,889 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,889 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:22,889 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:22,891 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,891 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:22,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:22,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:22,891 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:22,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:22,891 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:22,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:22,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:22,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:22,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:22,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69a3c856[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:22,892 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:22,893 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:22,893 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:22,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:22,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:22,893 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:22,893 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,893 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:22,893 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:22,893 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:22,894 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,894 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:22,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:22,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:22,894 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:22,894 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:22,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:22,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:22,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:22,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:22,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:22,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:22,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:22,897 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:22,897 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:22,897 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:22,897 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:22,897 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:22,898 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:22,898 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:22,898 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:22,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:22,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:22,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:22,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:22,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:22,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:22,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:22,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:22,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:22,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:22,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:22,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:22,900 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:22,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:22,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,901 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,908 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,908 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:22,908 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:22,908 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:22,908 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:22,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:22,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:22,908 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:22,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:22,908 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,908 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:22,908 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:22,908 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:22,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:22,908 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:22,908 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:22,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:22,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:22,927 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:22,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670642923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,927 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:22,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49118 deadline: 1680670642923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,927 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:22,927 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:22,927 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:22,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670642924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49098 deadline: 1680670642923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:22,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670642925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,008 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:23,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:23,011 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/fe0d8cc00b4c46e887bf34568f9018ed 2023-04-05 04:56:23,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:23,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:23,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:23,013 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:23,013 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:23,013 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:23,013 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:23,013 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,013 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:23,015 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,015 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:23,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:23,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:23,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:23,016 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,016 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:23,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:23,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:23,016 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:23,017 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:23,017 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:23,017 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:23,017 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:23,017 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:23,017 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:23,017 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:23,017 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:23,017 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:23,017 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:23,017 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:23,017 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:23,017 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:23,017 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,017 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:23,017 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:23,018 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,018 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:23,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:23,019 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,019 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:23,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:23,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:23,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:23,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:23,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:23,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68eb1558[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:23,020 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:23,020 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,022 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,022 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:23,022 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:23,022 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:23,023 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,023 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:23,025 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:23,025 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:23,025 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:23,025 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:23,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:23,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:23,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:23,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:23,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:23,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:23,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:23,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:23,027 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:23,028 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:23,028 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:23,028 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:23,028 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:23,029 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,029 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:23,029 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:23,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:23,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:23,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:23,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:23,030 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:23,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:23,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:23,034 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,034 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/717d10fece2f4089a1c4fa26e76802e2 2023-04-05 04:56:23,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:23,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:23,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:23,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:23,043 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:23,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:23,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:23,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:23,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:23,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:23,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:23,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:23,044 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,044 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,044 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:23,044 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:23,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:23,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:23,144 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:23,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:23,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:23,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:23,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:23,147 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:23,147 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:23,148 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:23,148 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:23,148 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,148 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:23,149 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,149 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:23,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:23,149 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,149 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:23,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:23,150 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:23,150 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:23,151 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:23,151 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:23,151 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:23,151 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:23,151 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:23,151 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:23,151 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:23,151 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:23,151 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:23,151 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:23,151 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:23,151 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:23,151 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,151 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:23,151 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:23,152 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,152 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:23,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:23,153 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,153 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:23,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:23,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:23,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:23,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:23,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:23,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fc0b9d8[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:23,155 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:23,155 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,156 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,156 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:23,156 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:23,156 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:23,156 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,156 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:23,157 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:23,157 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,157 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:23,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:23,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:23,157 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:23,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:23,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:23,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:23,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:23,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:23,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:23,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:23,160 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:23,160 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:23,160 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:23,160 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:23,160 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:23,163 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,163 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,163 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:23,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:23,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:23,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:23,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:23,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,166 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:23,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:23,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:23,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:23,171 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:23,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:23,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,179 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,179 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:23,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:23,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:23,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:23,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:23,179 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:23,179 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:23,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:23,179 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:23,179 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:23,179 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:23,179 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:23,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:23,179 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,180 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,180 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,180 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:23,230 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:23,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670643228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,230 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:23,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670643230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,231 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:23,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49118 deadline: 1680670643231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,232 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:23,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49098 deadline: 1680670643231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,232 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:23,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670643231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,279 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:23,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:23,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:23,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:23,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:23,284 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:23,284 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:23,284 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:23,284 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:23,284 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,284 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:23,285 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:23,285 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:23,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:23,285 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,285 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:23,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:23,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:23,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:23,286 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:23,286 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:23,286 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:23,286 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:23,286 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:23,286 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:23,286 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:23,286 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:23,286 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:23,287 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:23,287 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:23,287 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:23,287 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:23,287 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:23,287 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,287 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:23,287 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:23,288 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,291 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:23,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:23,291 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,291 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:23,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:23,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:23,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:23,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:23,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:23,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1200e322[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:23,294 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:23,294 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,301 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,301 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:23,302 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:23,302 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:23,302 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,302 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:23,304 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:23,305 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:23,305 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:23,305 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:23,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:23,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:23,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:23,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:23,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:23,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:23,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:23,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:23,308 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:23,308 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:23,308 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:23,308 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:23,308 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:23,312 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,312 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,312 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:23,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:23,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:23,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:23,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:23,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:23,313 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:23,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:23,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:23,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,315 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:23,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:23,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,317 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:23,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:23,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:23,317 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,317 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:23,318 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:23,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:23,318 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:23,318 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,318 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,318 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:23,318 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:23,318 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:23,318 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:23,318 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:23,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:23,418 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:23,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:23,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:23,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:23,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:23,423 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:23,423 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:23,423 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:23,423 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:23,424 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,424 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:23,425 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:23,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:23,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:23,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:23,425 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,426 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:23,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:23,426 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:23,426 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,426 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:23,430 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:23,430 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:23,430 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:23,430 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:23,430 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:23,442 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:23,442 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:23,448 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:23,448 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:23,448 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/2e9f0aece26c42f4a526739a3c1fe5ea 2023-04-05 04:56:23,448 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:23,449 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:23,449 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:23,449 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:23,449 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,449 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:23,449 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:23,451 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:23,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:23,452 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,456 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,456 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:23,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:23,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:23,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:23,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,463 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/fe0d8cc00b4c46e887bf34568f9018ed as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/fe0d8cc00b4c46e887bf34568f9018ed 2023-04-05 04:56:23,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:23,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:23,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6516784c[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:23,464 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:23,464 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,468 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,468 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:23,468 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:23,469 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:23,469 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,469 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:23,470 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:23,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:23,470 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:23,470 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:23,470 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:23,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:23,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:23,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:23,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:23,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:23,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,472 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/fe0d8cc00b4c46e887bf34568f9018ed, entries=250, sequenceid=213, filesize=16.4 K 2023-04-05 04:56:23,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:23,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:23,473 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:23,473 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:23,473 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:23,473 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:23,473 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:23,473 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/717d10fece2f4089a1c4fa26e76802e2 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/717d10fece2f4089a1c4fa26e76802e2 2023-04-05 04:56:23,475 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,475 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:23,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,475 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:23,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:23,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:23,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:23,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:23,477 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:23,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:23,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:23,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:23,479 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:23,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,481 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/717d10fece2f4089a1c4fa26e76802e2, entries=150, sequenceid=213, filesize=11.8 K 2023-04-05 04:56:23,482 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/2e9f0aece26c42f4a526739a3c1fe5ea as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/2e9f0aece26c42f4a526739a3c1fe5ea 2023-04-05 04:56:23,487 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,487 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:23,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:23,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:23,488 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:23,488 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:23,488 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:23,487 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:23,488 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:23,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:23,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:23,488 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:23,488 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,488 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,488 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,488 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:23,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:23,495 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/2e9f0aece26c42f4a526739a3c1fe5ea, entries=150, sequenceid=213, filesize=11.8 K 2023-04-05 04:56:23,497 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for ed7bbafaddb97066dca4b03f35c9eee7 in 899ms, sequenceid=213, compaction requested=true 2023-04-05 04:56:23,497 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:23,497 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:56:23,497 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:56:23,497 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:56:23,497 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-05 04:56:23,498 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41298 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:56:23,499 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): ed7bbafaddb97066dca4b03f35c9eee7/A is initiating minor compaction (all files) 2023-04-05 04:56:23,499 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ed7bbafaddb97066dca4b03f35c9eee7/A in TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:23,499 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/bc8a097904dc44dab24e1828eb2a9dc7, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/6cb305b077864af48242ff2c437b7121, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/fe0d8cc00b4c46e887bf34568f9018ed] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp, totalSize=40.3 K 2023-04-05 04:56:23,500 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:56:23,502 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting bc8a097904dc44dab24e1828eb2a9dc7, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1680670581043 2023-04-05 04:56:23,503 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 6cb305b077864af48242ff2c437b7121, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1680670581152 2023-04-05 04:56:23,504 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting fe0d8cc00b4c46e887bf34568f9018ed, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1680670582285 2023-04-05 04:56:23,505 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36518 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:56:23,505 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): ed7bbafaddb97066dca4b03f35c9eee7/B is initiating minor compaction (all files) 2023-04-05 04:56:23,505 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ed7bbafaddb97066dca4b03f35c9eee7/B in TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:23,505 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/1df758f917c64c86824b85e2284178de, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/f75054eb216948b8a5812b01ae9a11f7, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/717d10fece2f4089a1c4fa26e76802e2] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp, totalSize=35.7 K 2023-04-05 04:56:23,506 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 1df758f917c64c86824b85e2284178de, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1680670581043 2023-04-05 04:56:23,506 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting f75054eb216948b8a5812b01ae9a11f7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1680670581152 2023-04-05 04:56:23,508 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 717d10fece2f4089a1c4fa26e76802e2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1680670582288 2023-04-05 04:56:23,522 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): ed7bbafaddb97066dca4b03f35c9eee7#A#compaction#45 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:56:23,523 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): ed7bbafaddb97066dca4b03f35c9eee7#B#compaction#46 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:56:23,588 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:23,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:23,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:23,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:23,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:23,598 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:23,598 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:23,598 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:23,615 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:23,616 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,616 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:23,617 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,621 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,621 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:23,622 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:23,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:23,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:23,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:23,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:23,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:23,623 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:23,624 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:23,624 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:23,624 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:23,624 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:23,627 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:23,627 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:23,627 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:23,627 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:23,627 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ed7bbafaddb97066dca4b03f35c9eee7 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-05 04:56:23,627 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=A 2023-04-05 04:56:23,627 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:23,628 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=B 2023-04-05 04:56:23,628 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:23,628 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=C 2023-04-05 04:56:23,628 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:23,673 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/9e60b08dc6f04d90b2225cbc01722dd9 2023-04-05 04:56:23,709 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/220c5f4393e44d438cf105e56728f37a 2023-04-05 04:56:23,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:23,735 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:23,752 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:23,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49098 deadline: 1680670643745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:23,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670643745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:23,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670643746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,754 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:23,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49118 deadline: 1680670643748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:23,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670643754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,766 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/f3a5d04f8ea7482ead6e0e85b07928ad 2023-04-05 04:56:23,780 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/9e60b08dc6f04d90b2225cbc01722dd9 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/9e60b08dc6f04d90b2225cbc01722dd9 2023-04-05 04:56:23,788 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/9e60b08dc6f04d90b2225cbc01722dd9, entries=150, sequenceid=233, filesize=11.8 K 2023-04-05 04:56:23,792 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/220c5f4393e44d438cf105e56728f37a as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/220c5f4393e44d438cf105e56728f37a 2023-04-05 04:56:23,800 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/220c5f4393e44d438cf105e56728f37a, entries=150, sequenceid=233, filesize=11.8 K 2023-04-05 04:56:23,806 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/f3a5d04f8ea7482ead6e0e85b07928ad as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/f3a5d04f8ea7482ead6e0e85b07928ad 2023-04-05 04:56:23,814 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/f3a5d04f8ea7482ead6e0e85b07928ad, entries=150, sequenceid=233, filesize=11.8 K 2023-04-05 04:56:23,815 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for ed7bbafaddb97066dca4b03f35c9eee7 in 188ms, sequenceid=233, compaction requested=true 2023-04-05 04:56:23,816 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:23,816 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:23,816 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:23,816 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:23,816 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,816 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:23,816 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:23,818 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,818 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:23,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:23,819 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,819 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:23,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:23,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:23,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:23,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:23,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:23,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5567f063[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:23,820 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:23,821 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,821 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,822 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:23,822 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:23,822 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:23,822 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,822 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:23,823 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:23,823 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,823 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:23,823 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:23,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:23,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:23,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:23,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:23,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:23,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:23,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:23,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:23,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:23,844 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:23,844 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:23,844 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:23,844 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:23,844 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:23,845 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,845 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:23,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:23,845 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:23,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:23,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:23,846 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:23,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:23,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:23,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:23,847 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:23,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:23,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,849 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,849 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:23,849 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:23,849 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:23,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:23,850 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:23,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:23,850 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:23,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:23,850 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,850 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,850 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:23,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:23,850 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:23,850 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:23,850 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:23,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:23,856 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ed7bbafaddb97066dca4b03f35c9eee7 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-05 04:56:23,857 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=A 2023-04-05 04:56:23,857 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:23,857 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=B 2023-04-05 04:56:23,857 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:23,857 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=C 2023-04-05 04:56:23,857 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:23,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:23,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:23,894 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:23,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49118 deadline: 1680670643885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49098 deadline: 1680670643885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,895 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:23,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670643887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,899 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:23,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670643894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:23,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670643894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,901 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/9fd030343e40449bbf456aa41a03df63 2023-04-05 04:56:23,920 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/3a97ec790d354ff4b0f1a44b15a59bc1 2023-04-05 04:56:23,950 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:23,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:23,953 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/9ba8866d9ac7433c97f587502ebac0cd 2023-04-05 04:56:23,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:23,960 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/1ef80b8838904694b9f3ededc4b4db58 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/1ef80b8838904694b9f3ededc4b4db58 2023-04-05 04:56:23,965 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/9fd030343e40449bbf456aa41a03df63 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/9fd030343e40449bbf456aa41a03df63 2023-04-05 04:56:23,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:23,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:23,966 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:23,966 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:23,966 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:23,966 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:23,967 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,967 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:23,967 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:23,968 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:23,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:23,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:23,968 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,968 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:23,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:23,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:23,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:23,969 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:23,971 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ed7bbafaddb97066dca4b03f35c9eee7/A of ed7bbafaddb97066dca4b03f35c9eee7 into 1ef80b8838904694b9f3ededc4b4db58(size=12.3 K), total size for store is 24.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:56:23,971 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:23,971 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7., storeName=ed7bbafaddb97066dca4b03f35c9eee7/A, priority=13, startTime=1680670583497; duration=0sec 2023-04-05 04:56:23,971 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:56:23,971 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:56:23,973 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/9fd030343e40449bbf456aa41a03df63, entries=200, sequenceid=246, filesize=14.1 K 2023-04-05 04:56:23,974 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:23,974 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:23,974 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:23,974 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:23,974 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:23,974 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/3a97ec790d354ff4b0f1a44b15a59bc1 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/3a97ec790d354ff4b0f1a44b15a59bc1 2023-04-05 04:56:23,975 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48554 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:56:23,975 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): ed7bbafaddb97066dca4b03f35c9eee7/C is initiating minor compaction (all files) 2023-04-05 04:56:23,975 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ed7bbafaddb97066dca4b03f35c9eee7/C in TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:23,975 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/9ae8be7063fd4972a5def3b473d7f4f2, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/798aff70a68e4034aad6d62b194ec086, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/2e9f0aece26c42f4a526739a3c1fe5ea, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/f3a5d04f8ea7482ead6e0e85b07928ad] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp, totalSize=47.4 K 2023-04-05 04:56:23,976 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 9ae8be7063fd4972a5def3b473d7f4f2, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1680670581043 2023-04-05 04:56:23,976 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 798aff70a68e4034aad6d62b194ec086, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1680670581152 2023-04-05 04:56:23,977 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 2e9f0aece26c42f4a526739a3c1fe5ea, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1680670582288 2023-04-05 04:56:23,978 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting f3a5d04f8ea7482ead6e0e85b07928ad, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1680670582609 2023-04-05 04:56:23,986 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/589f3fba331e47449f8f5d6781db8d94 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/589f3fba331e47449f8f5d6781db8d94 2023-04-05 04:56:23,992 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): ed7bbafaddb97066dca4b03f35c9eee7#C#compaction#53 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:56:23,994 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:23,994 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:23,994 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:23,994 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:23,994 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:23,994 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:23,994 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:23,994 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:23,994 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,995 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ed7bbafaddb97066dca4b03f35c9eee7/B of ed7bbafaddb97066dca4b03f35c9eee7 into 589f3fba331e47449f8f5d6781db8d94(size=12.3 K), total size for store is 24.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:56:23,995 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:23,995 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7., storeName=ed7bbafaddb97066dca4b03f35c9eee7/B, priority=13, startTime=1680670583497; duration=0sec 2023-04-05 04:56:23,995 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:56:23,998 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:23,998 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:23,999 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,999 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:23,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:23,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:23,999 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:23,999 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:24,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:24,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:24,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:24,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:24,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:24,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:24,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f539d6e[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:24,001 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:24,001 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:24,002 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:24,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:24,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:24,003 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:24,003 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:24,003 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:24,003 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:24,003 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:24,004 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:24,004 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:24,004 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:24,004 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:24,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:24,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:24,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:24,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:24,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:24,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:24,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:24,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:24,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:24,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:24,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:24,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:24,009 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:24,010 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:24,010 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:24,010 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:24,010 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:24,012 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:24,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49118 deadline: 1680670643996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:24,013 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:24,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49098 deadline: 1680670643996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:24,015 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:24,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670644002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:24,016 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/3a97ec790d354ff4b0f1a44b15a59bc1, entries=150, sequenceid=246, filesize=11.8 K 2023-04-05 04:56:24,016 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:24,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670644002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:24,017 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:24,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670644002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:24,017 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/9ba8866d9ac7433c97f587502ebac0cd as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/9ba8866d9ac7433c97f587502ebac0cd 2023-04-05 04:56:24,019 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:24,019 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:24,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:24,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:24,020 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:24,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:24,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:24,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:24,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:24,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:24,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:24,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:24,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:24,021 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:24,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:24,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:24,023 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:24,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:24,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:24,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:24,027 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/9ba8866d9ac7433c97f587502ebac0cd, entries=150, sequenceid=246, filesize=11.8 K 2023-04-05 04:56:24,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:24,028 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:24,028 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:24,028 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:24,028 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:24,028 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:24,028 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:24,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:24,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:24,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:24,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:24,028 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for ed7bbafaddb97066dca4b03f35c9eee7 in 172ms, sequenceid=246, compaction requested=true 2023-04-05 04:56:24,028 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:24,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:24,028 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:24,028 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:24,028 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:24,029 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:24,029 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:24,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:24,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:24,029 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:24,029 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:56:24,029 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:56:24,029 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:56:24,029 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-05 04:56:24,030 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39010 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:56:24,030 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): ed7bbafaddb97066dca4b03f35c9eee7/A is initiating minor compaction (all files) 2023-04-05 04:56:24,030 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ed7bbafaddb97066dca4b03f35c9eee7/A in TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:24,030 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/1ef80b8838904694b9f3ededc4b4db58, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/9e60b08dc6f04d90b2225cbc01722dd9, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/9fd030343e40449bbf456aa41a03df63] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp, totalSize=38.1 K 2023-04-05 04:56:24,031 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 1ef80b8838904694b9f3ededc4b4db58, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1680670582288 2023-04-05 04:56:24,032 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 9e60b08dc6f04d90b2225cbc01722dd9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1680670582609 2023-04-05 04:56:24,032 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 9fd030343e40449bbf456aa41a03df63, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1680670583742 2023-04-05 04:56:24,054 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/b80dab0cb2ae4152ab07e01fc8248a5a as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/b80dab0cb2ae4152ab07e01fc8248a5a 2023-04-05 04:56:24,059 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): ed7bbafaddb97066dca4b03f35c9eee7#A#compaction#54 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:56:24,067 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ed7bbafaddb97066dca4b03f35c9eee7/C of ed7bbafaddb97066dca4b03f35c9eee7 into b80dab0cb2ae4152ab07e01fc8248a5a(size=12.3 K), total size for store is 24.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:56:24,067 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:24,067 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7., storeName=ed7bbafaddb97066dca4b03f35c9eee7/C, priority=12, startTime=1680670583497; duration=0sec 2023-04-05 04:56:24,068 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-05 04:56:24,068 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:56:24,070 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36620 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:56:24,070 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): ed7bbafaddb97066dca4b03f35c9eee7/B is initiating minor compaction (all files) 2023-04-05 04:56:24,070 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ed7bbafaddb97066dca4b03f35c9eee7/B in TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:24,070 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/589f3fba331e47449f8f5d6781db8d94, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/220c5f4393e44d438cf105e56728f37a, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/3a97ec790d354ff4b0f1a44b15a59bc1] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp, totalSize=35.8 K 2023-04-05 04:56:24,072 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 589f3fba331e47449f8f5d6781db8d94, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1680670582288 2023-04-05 04:56:24,073 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 220c5f4393e44d438cf105e56728f37a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1680670582609 2023-04-05 04:56:24,073 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 3a97ec790d354ff4b0f1a44b15a59bc1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1680670583742 2023-04-05 04:56:24,095 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): ed7bbafaddb97066dca4b03f35c9eee7#B#compaction#55 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:56:24,129 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:24,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:24,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:24,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:24,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:24,140 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:24,140 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:24,140 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:24,140 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:24,141 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:24,141 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:24,142 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:24,142 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:24,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:24,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:24,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:24,142 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:24,143 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:24,143 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:24,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:24,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:24,143 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:24,154 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:24,154 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:24,154 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:24,154 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:24,154 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:24,154 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:24,155 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:24,155 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:24,155 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:24,155 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ed7bbafaddb97066dca4b03f35c9eee7 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-05 04:56:24,155 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=A 2023-04-05 04:56:24,155 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:24,155 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=B 2023-04-05 04:56:24,156 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:24,156 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=C 2023-04-05 04:56:24,156 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:24,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:24,217 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:24,228 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:24,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670644223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:24,228 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:24,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670644224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:24,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:24,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49098 deadline: 1680670644226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:24,230 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:24,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49118 deadline: 1680670644228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:24,230 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:24,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670644228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:24,329 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:24,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670644329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:24,329 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:24,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670644329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:24,332 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:24,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49098 deadline: 1680670644331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:24,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:24,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49118 deadline: 1680670644335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:24,338 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:24,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670644337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:24,504 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/6b08b8b81e4745278966b3ea4cf33e9a as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/6b08b8b81e4745278966b3ea4cf33e9a 2023-04-05 04:56:24,510 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ed7bbafaddb97066dca4b03f35c9eee7/A of ed7bbafaddb97066dca4b03f35c9eee7 into 6b08b8b81e4745278966b3ea4cf33e9a(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:56:24,510 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:24,510 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7., storeName=ed7bbafaddb97066dca4b03f35c9eee7/A, priority=13, startTime=1680670584029; duration=0sec 2023-04-05 04:56:24,510 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:56:24,510 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-05 04:56:24,512 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-05 04:56:24,512 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-05 04:56:24,512 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. because compaction request was cancelled 2023-04-05 04:56:24,532 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:24,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670644531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:24,536 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:24,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49098 deadline: 1680670644535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:24,538 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:24,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670644538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:24,539 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:24,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49118 deadline: 1680670644539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:24,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:24,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670644541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:24,546 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/9864b9f055044843b16a6ad6bd0179fd as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/9864b9f055044843b16a6ad6bd0179fd 2023-04-05 04:56:24,558 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ed7bbafaddb97066dca4b03f35c9eee7/B of ed7bbafaddb97066dca4b03f35c9eee7 into 9864b9f055044843b16a6ad6bd0179fd(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:56:24,558 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:24,558 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7., storeName=ed7bbafaddb97066dca4b03f35c9eee7/B, priority=13, startTime=1680670584029; duration=0sec 2023-04-05 04:56:24,558 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:56:24,593 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/13fecde8a5de48d6a9eaa4117052cc43 2023-04-05 04:56:24,613 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/e47286b227bc4e0db8b4c4e24fe3c14d 2023-04-05 04:56:24,834 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:24,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49082 deadline: 1680670644834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:24,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:24,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49098 deadline: 1680670644838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:24,843 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:24,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670644842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:24,843 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:24,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49118 deadline: 1680670644843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:24,848 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:24,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670644847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,118 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/b83ee6fe57214905a966135ddff1a7cc 2023-04-05 04:56:25,126 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/13fecde8a5de48d6a9eaa4117052cc43 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/13fecde8a5de48d6a9eaa4117052cc43 2023-04-05 04:56:25,136 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/13fecde8a5de48d6a9eaa4117052cc43, entries=150, sequenceid=268, filesize=11.9 K 2023-04-05 04:56:25,137 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/e47286b227bc4e0db8b4c4e24fe3c14d as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/e47286b227bc4e0db8b4c4e24fe3c14d 2023-04-05 04:56:25,156 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/e47286b227bc4e0db8b4c4e24fe3c14d, entries=150, sequenceid=268, filesize=11.9 K 2023-04-05 04:56:25,157 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/b83ee6fe57214905a966135ddff1a7cc as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/b83ee6fe57214905a966135ddff1a7cc 2023-04-05 04:56:25,166 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/b83ee6fe57214905a966135ddff1a7cc, entries=150, sequenceid=268, filesize=11.9 K 2023-04-05 04:56:25,176 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for ed7bbafaddb97066dca4b03f35c9eee7 in 1020ms, sequenceid=268, compaction requested=true 2023-04-05 04:56:25,176 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:25,176 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:25,176 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:25,176 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:25,176 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,176 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:25,176 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:25,177 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,177 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:25,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:25,178 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,178 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:25,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:25,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:25,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:25,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:25,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:25,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a5f6a55[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:25,180 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:25,180 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,181 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,181 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:25,181 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:25,181 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:25,181 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,181 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:25,182 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:25,182 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:25,182 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:25,182 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:25,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:25,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:25,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:25,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:25,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:25,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:25,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:25,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:25,185 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:25,185 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:25,185 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:25,185 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:25,185 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:25,186 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,186 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:25,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:25,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,187 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:25,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,187 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:25,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:25,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:25,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:25,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:25,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:25,188 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:25,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:25,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,190 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:25,190 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:25,190 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,190 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:25,191 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:25,191 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,191 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,191 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,191 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:25,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:25,192 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:25,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:25,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:25,193 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:25,193 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:25,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:25,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:25,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:25,293 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:25,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:25,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:25,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:25,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:25,296 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:25,296 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:25,296 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:25,296 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:25,297 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,297 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:25,297 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:25,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:25,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:25,297 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:25,298 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,298 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:25,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:25,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:25,298 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:25,298 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:25,299 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:25,299 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:25,299 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:25,299 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:25,302 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:25,302 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:25,302 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:25,302 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:25,302 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ed7bbafaddb97066dca4b03f35c9eee7 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-05 04:56:25,303 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=A 2023-04-05 04:56:25,303 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:25,303 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=B 2023-04-05 04:56:25,303 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:25,303 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=C 2023-04-05 04:56:25,303 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:25,318 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/20aa8e818948457b9d56874a17efeaef 2023-04-05 04:56:25,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:25,343 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:25,357 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/25b5f2ef18234268ab6525e68f41ee9a 2023-04-05 04:56:25,372 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/321f0b3dd8774c42a5f092dc5d57ca14 2023-04-05 04:56:25,378 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/20aa8e818948457b9d56874a17efeaef as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/20aa8e818948457b9d56874a17efeaef 2023-04-05 04:56:25,384 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/20aa8e818948457b9d56874a17efeaef, entries=150, sequenceid=283, filesize=11.9 K 2023-04-05 04:56:25,385 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/25b5f2ef18234268ab6525e68f41ee9a as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/25b5f2ef18234268ab6525e68f41ee9a 2023-04-05 04:56:25,393 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:25,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49098 deadline: 1680670645383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,395 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/25b5f2ef18234268ab6525e68f41ee9a, entries=150, sequenceid=283, filesize=11.9 K 2023-04-05 04:56:25,397 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/321f0b3dd8774c42a5f092dc5d57ca14 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/321f0b3dd8774c42a5f092dc5d57ca14 2023-04-05 04:56:25,399 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:25,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49082 deadline: 1680670645390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,399 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:25,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670645392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,400 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:25,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670645392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,400 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:25,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49118 deadline: 1680670645394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,411 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/321f0b3dd8774c42a5f092dc5d57ca14, entries=150, sequenceid=283, filesize=11.9 K 2023-04-05 04:56:25,414 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for ed7bbafaddb97066dca4b03f35c9eee7 in 112ms, sequenceid=283, compaction requested=true 2023-04-05 04:56:25,414 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:25,414 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:25,414 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:25,414 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:25,414 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,422 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:25,422 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:25,423 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,423 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:25,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:25,423 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,424 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:25,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:25,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:25,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:25,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:25,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:25,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cddb94a[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:25,425 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:25,426 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,426 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,427 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:25,427 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:25,427 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:25,427 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,427 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:25,427 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:25,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:25,428 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:25,428 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:25,428 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:25,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:25,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:25,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:25,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:25,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:25,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:25,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:25,431 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:25,431 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:25,431 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:25,431 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:25,431 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:25,432 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,432 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:25,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,432 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:25,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:25,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:25,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:25,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:25,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:25,434 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:25,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:25,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,436 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:25,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:25,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,445 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,445 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:25,445 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:25,445 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:25,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:25,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:25,446 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,446 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,446 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:25,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:25,445 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:25,446 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:25,446 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:25,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:25,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:25,446 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:25,446 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:25,502 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670585370, max timestamp=1680670585495Num uniques 859; 2023-04-05 04:56:25,502 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:56:25,502 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ed7bbafaddb97066dca4b03f35c9eee7 3/3 column families, dataSize=89.45 KB heapSize=262.11 KB 2023-04-05 04:56:25,502 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=A 2023-04-05 04:56:25,502 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:56:25,502 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=B 2023-04-05 04:56:25,503 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:56:25,503 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=C 2023-04-05 04:56:25,503 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:56:25,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:25,517 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/4d0ab75fc6384108b66cf9b0e23aa4cb 2023-04-05 04:56:25,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:25,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49098 deadline: 1680670645518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:25,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49112 deadline: 1680670645524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,530 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:25,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:25,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49082 deadline: 1680670645524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670645525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,531 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:25,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49118 deadline: 1680670645526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,546 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:25,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:25,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:25,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:25,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:25,552 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:25,552 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:25,552 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:25,552 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:25,553 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,553 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:25,554 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:25,554 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:25,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:25,554 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,554 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:25,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:25,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:25,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:25,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:25,557 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:25,557 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:25,557 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:25,557 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:25,557 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:25,561 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:25,561 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:25,561 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:25,561 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:25,561 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:25,561 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:25,561 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:25,561 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:25,562 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,562 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:25,562 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:25,563 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/2ad0fde45b8547b9aa03cf6e685f965d 2023-04-05 04:56:25,568 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,568 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:25,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:25,568 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,568 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:25,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:25,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:25,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:25,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:25,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:25,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46d3a05a[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:25,570 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:25,570 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,571 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,571 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:25,571 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:25,571 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:25,571 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,571 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:25,573 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:25,573 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:25,573 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:25,573 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:25,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:25,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:25,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:25,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:25,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:25,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:25,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:25,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:25,577 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:25,577 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:25,577 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:25,577 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:25,577 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:25,584 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,584 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:25,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,585 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,585 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:25,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:25,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:25,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:25,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:25,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,594 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,597 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,597 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:25,597 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:25,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:25,597 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:25,597 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:25,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:25,597 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:25,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:25,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:25,597 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,597 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,598 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,598 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,598 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:25,598 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:25,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:25,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:25,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:25,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49098 deadline: 1680670645627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,632 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:25,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49112 deadline: 1680670645631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,633 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:25,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49082 deadline: 1680670645632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:25,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:25,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49118 deadline: 1680670645635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49154 deadline: 1680670645635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,698 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:25,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:25,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:25,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:25,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:25,702 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:25,702 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:25,702 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:25,702 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:25,704 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,704 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:25,705 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,705 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:25,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:25,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:25,705 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,705 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:25,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:25,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:25,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:25,706 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:25,707 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:25,707 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:25,707 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:25,707 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:25,707 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:25,707 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:25,707 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:25,707 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:25,707 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:25,707 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:25,707 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:25,707 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:25,707 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:25,707 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,707 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:25,707 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:25,708 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,709 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,709 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:25,709 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:25,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:25,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:25,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:25,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:25,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:25,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:25,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3518fd03[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:25,711 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:25,711 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,712 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,712 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:25,715 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,715 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:25,715 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:25,715 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:25,718 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,718 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:25,718 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:25,718 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:25,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:25,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:25,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:25,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:25,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:25,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:25,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:25,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:25,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:25,720 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:25,720 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:25,721 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:25,721 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:25,721 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:25,722 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,722 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:25,722 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:25,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:25,723 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,724 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,727 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,727 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:25,727 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:25,727 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,727 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,727 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,727 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,727 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:25,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:25,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:25,727 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:25,727 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:25,730 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:25,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:25,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:25,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:25,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:25,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:25,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:25,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:25,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:25,828 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:25,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:25,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:25,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:25,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:25,831 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:25,831 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:25,831 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:25,831 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:25,831 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,831 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:25,832 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,832 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:25,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:25,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:25,832 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:25,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:25,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49098 deadline: 1680670645831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,832 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:25,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:25,833 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:25,833 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:25,833 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:25,833 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,833 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:25,833 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:25,833 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:25,833 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:25,833 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:25,833 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:25,833 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:25,834 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:25,834 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:25,834 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:25,834 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:25,834 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:25,834 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,834 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:25,834 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:25,835 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,835 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:25,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:25,835 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,835 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:25,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:25,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:25,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:25,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:25,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:25,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34143437[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:25,836 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:25,836 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:25,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49112 deadline: 1680670645835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:25,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49082 deadline: 1680670645835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,844 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:25,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49118 deadline: 1680670645844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,845 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:25,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49154 deadline: 1680670645844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,846 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,846 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:25,846 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:25,846 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:25,846 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,846 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:25,854 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,854 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:25,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:25,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:25,854 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:25,854 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:25,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:25,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:25,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:25,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:25,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:25,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:25,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:25,856 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:25,856 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:25,856 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:25,856 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:25,856 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:25,860 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,860 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,861 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:25,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:25,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:25,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:25,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:25,861 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:25,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:25,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:25,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:25,871 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:25,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:25,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,886 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,886 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:25,886 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:25,886 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,886 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,886 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:25,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:25,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:25,886 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:25,887 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:25,886 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:25,887 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:25,887 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:25,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:25,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:25,887 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:25,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:25,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:25,985 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/ffeb9c2d4e834763ba4be2739e108c35 2023-04-05 04:56:25,987 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:25,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:25,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:25,992 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/4d0ab75fc6384108b66cf9b0e23aa4cb as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/4d0ab75fc6384108b66cf9b0e23aa4cb 2023-04-05 04:56:25,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:25,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:25,992 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:25,992 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:25,992 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:25,992 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:25,993 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,993 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:25,994 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:25,994 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:25,994 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:25,994 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:25,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:25,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:25,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:25,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:25,995 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:25,995 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:25,995 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:25,996 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:25,996 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:25,999 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/4d0ab75fc6384108b66cf9b0e23aa4cb, entries=150, sequenceid=304, filesize=11.9 K 2023-04-05 04:56:26,000 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/2ad0fde45b8547b9aa03cf6e685f965d as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/2ad0fde45b8547b9aa03cf6e685f965d 2023-04-05 04:56:26,001 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:26,001 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:26,001 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:26,001 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:26,001 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:26,001 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:26,001 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:26,001 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:26,001 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,001 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:26,001 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:26,009 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/2ad0fde45b8547b9aa03cf6e685f965d, entries=150, sequenceid=304, filesize=11.9 K 2023-04-05 04:56:26,010 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/ffeb9c2d4e834763ba4be2739e108c35 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/ffeb9c2d4e834763ba4be2739e108c35 2023-04-05 04:56:26,012 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,012 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:26,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:26,014 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:26,014 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:26,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:26,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:26,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:26,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:26,015 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:26,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7542cc16[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:26,015 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,018 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,018 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:26,018 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:26,018 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,018 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:26,018 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:26,019 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:26,019 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,019 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:26,019 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:26,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:26,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:26,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:26,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:26,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:26,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:26,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:26,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:26,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:26,022 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:26,022 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:26,022 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:26,022 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:26,022 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:26,024 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,024 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:26,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,024 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:26,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:26,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:26,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:26,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:26,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:26,027 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,027 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/ffeb9c2d4e834763ba4be2739e108c35, entries=150, sequenceid=304, filesize=11.9 K 2023-04-05 04:56:26,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:26,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:26,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,028 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:26,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:26,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,030 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~89.45 KB/91600, heapSize ~262.06 KB/268352, currentSize=93.93 KB/96180 for ed7bbafaddb97066dca4b03f35c9eee7 in 528ms, sequenceid=304, compaction requested=true 2023-04-05 04:56:26,030 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:26,030 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:56:26,030 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:56:26,031 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:56:26,031 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:56:26,031 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:56:26,032 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49208 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:56:26,032 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): ed7bbafaddb97066dca4b03f35c9eee7/A is initiating minor compaction (all files) 2023-04-05 04:56:26,032 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ed7bbafaddb97066dca4b03f35c9eee7/A in TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:26,032 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/6b08b8b81e4745278966b3ea4cf33e9a, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/13fecde8a5de48d6a9eaa4117052cc43, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/20aa8e818948457b9d56874a17efeaef, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/4d0ab75fc6384108b66cf9b0e23aa4cb] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp, totalSize=48.1 K 2023-04-05 04:56:26,033 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 6b08b8b81e4745278966b3ea4cf33e9a, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1680670583742 2023-04-05 04:56:26,033 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49208 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:56:26,033 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 13fecde8a5de48d6a9eaa4117052cc43, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1680670583878 2023-04-05 04:56:26,033 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): ed7bbafaddb97066dca4b03f35c9eee7/B is initiating minor compaction (all files) 2023-04-05 04:56:26,034 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ed7bbafaddb97066dca4b03f35c9eee7/B in TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:26,034 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/9864b9f055044843b16a6ad6bd0179fd, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/e47286b227bc4e0db8b4c4e24fe3c14d, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/25b5f2ef18234268ab6525e68f41ee9a, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/2ad0fde45b8547b9aa03cf6e685f965d] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp, totalSize=48.1 K 2023-04-05 04:56:26,035 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,035 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:26,035 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:26,035 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,035 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,035 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,035 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:26,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:26,035 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:26,036 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:26,036 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:26,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:26,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:26,036 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 20aa8e818948457b9d56874a17efeaef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1680670584225 2023-04-05 04:56:26,036 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:26,036 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:26,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:26,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:26,037 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 4d0ab75fc6384108b66cf9b0e23aa4cb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1680670585370 2023-04-05 04:56:26,037 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 9864b9f055044843b16a6ad6bd0179fd, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1680670583742 2023-04-05 04:56:26,038 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting e47286b227bc4e0db8b4c4e24fe3c14d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1680670583878 2023-04-05 04:56:26,038 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 25b5f2ef18234268ab6525e68f41ee9a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1680670584225 2023-04-05 04:56:26,039 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 2ad0fde45b8547b9aa03cf6e685f965d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1680670585370 2023-04-05 04:56:26,050 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): ed7bbafaddb97066dca4b03f35c9eee7#A#compaction#65 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:56:26,055 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): ed7bbafaddb97066dca4b03f35c9eee7#B#compaction#66 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:56:26,091 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/e464dfa7666943379ae0b788e3315c54 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/e464dfa7666943379ae0b788e3315c54 2023-04-05 04:56:26,093 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/d04cc117816545fea7621d1ab431aa53 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/d04cc117816545fea7621d1ab431aa53 2023-04-05 04:56:26,100 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ed7bbafaddb97066dca4b03f35c9eee7/A of ed7bbafaddb97066dca4b03f35c9eee7 into e464dfa7666943379ae0b788e3315c54(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:56:26,101 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:26,101 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7., storeName=ed7bbafaddb97066dca4b03f35c9eee7/A, priority=12, startTime=1680670586030; duration=0sec 2023-04-05 04:56:26,101 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:56:26,101 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 04:56:26,104 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61176 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 04:56:26,105 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): ed7bbafaddb97066dca4b03f35c9eee7/C is initiating minor compaction (all files) 2023-04-05 04:56:26,105 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ed7bbafaddb97066dca4b03f35c9eee7/C in TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:26,105 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/b80dab0cb2ae4152ab07e01fc8248a5a, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/9ba8866d9ac7433c97f587502ebac0cd, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/b83ee6fe57214905a966135ddff1a7cc, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/321f0b3dd8774c42a5f092dc5d57ca14, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/ffeb9c2d4e834763ba4be2739e108c35] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp, totalSize=59.7 K 2023-04-05 04:56:26,105 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting b80dab0cb2ae4152ab07e01fc8248a5a, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1680670582609 2023-04-05 04:56:26,106 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 9ba8866d9ac7433c97f587502ebac0cd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1680670583742 2023-04-05 04:56:26,107 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting b83ee6fe57214905a966135ddff1a7cc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1680670583878 2023-04-05 04:56:26,107 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 321f0b3dd8774c42a5f092dc5d57ca14, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1680670584225 2023-04-05 04:56:26,108 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ed7bbafaddb97066dca4b03f35c9eee7/B of ed7bbafaddb97066dca4b03f35c9eee7 into d04cc117816545fea7621d1ab431aa53(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:56:26,108 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:26,108 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7., storeName=ed7bbafaddb97066dca4b03f35c9eee7/B, priority=12, startTime=1680670586031; duration=0sec 2023-04-05 04:56:26,108 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:56:26,109 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting ffeb9c2d4e834763ba4be2739e108c35, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1680670585370 2023-04-05 04:56:26,124 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): ed7bbafaddb97066dca4b03f35c9eee7#C#compaction#67 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:56:26,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:26,135 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ed7bbafaddb97066dca4b03f35c9eee7 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-05 04:56:26,136 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=A 2023-04-05 04:56:26,136 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:26,136 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=B 2023-04-05 04:56:26,136 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:26,136 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=C 2023-04-05 04:56:26,136 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:26,136 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:26,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:26,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:26,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:26,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:26,143 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:26,143 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:26,143 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:26,143 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:26,144 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,144 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:26,145 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:26,145 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:26,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:26,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:26,145 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,145 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:26,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:26,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:26,146 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:26,154 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:26,154 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:26,154 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:26,154 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:26,154 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:26,155 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:26,155 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:26,155 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:26,155 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:26,155 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:26,155 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:26,155 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:26,155 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:26,155 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,155 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:26,155 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:26,165 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:26,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49082 deadline: 1680670646155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,166 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:26,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49154 deadline: 1680670646160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,167 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:26,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49112 deadline: 1680670646161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,167 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:26,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49118 deadline: 1680670646161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,168 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,168 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:26,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:26,168 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,168 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:26,169 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:26,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49098 deadline: 1680670646163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:26,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:26,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:26,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:26,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:26,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e486394[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:26,175 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:26,175 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,178 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,178 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:26,178 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:26,178 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:26,178 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,179 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:26,179 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,179 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:26,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:26,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:26,179 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:26,180 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:26,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:26,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:26,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:26,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:26,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:26,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:26,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:26,187 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:26,187 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:26,187 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:26,187 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:26,188 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:26,193 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,193 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:26,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,193 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:26,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:26,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:26,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:26,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:26,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,194 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:26,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:26,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:26,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,195 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:26,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:26,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,204 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,204 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:26,204 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:26,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:26,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:26,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:26,204 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:26,204 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,204 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,204 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,204 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,204 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:26,204 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:26,204 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:26,204 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:26,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:26,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:26,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:26,267 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:26,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49082 deadline: 1680670646266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,272 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:26,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49154 deadline: 1680670646267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,272 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:26,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49112 deadline: 1680670646268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,273 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:26,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49118 deadline: 1680670646268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,274 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:26,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49098 deadline: 1680670646270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,304 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:26,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:26,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:26,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:26,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:26,322 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:26,322 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:26,322 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:26,322 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:26,323 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,323 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:26,323 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:26,323 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:26,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:26,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:26,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,325 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,325 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:26,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:26,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:26,331 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:26,331 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:26,331 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:26,331 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:26,331 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:26,331 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:26,332 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:26,332 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:26,332 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:26,332 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:26,332 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:26,332 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:26,332 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:26,332 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:26,332 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,332 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:26,332 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:26,337 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:26,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:26,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:26,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:26,338 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:26,338 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,338 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:26,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:26,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:26,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c4a63bb[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:26,339 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:26,339 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,343 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,344 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:26,344 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:26,344 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:26,344 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,344 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:26,345 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:26,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:26,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:26,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:26,346 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:26,346 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:26,346 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:26,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:26,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:26,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:26,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:26,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:26,347 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:26,347 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:26,347 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:26,348 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:26,348 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:26,351 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,351 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:26,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,352 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:26,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:26,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:26,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:26,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:26,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,353 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:26,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:26,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:26,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,354 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:26,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:26,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,356 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,356 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:26,356 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:26,356 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:26,356 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:26,356 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,356 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,356 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:26,356 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:26,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:26,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:26,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:26,356 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,357 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:26,357 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:26,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:26,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:26,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:26,457 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:26,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:26,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:26,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:26,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49082 deadline: 1680670646469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:26,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:26,473 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:26,473 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:26,474 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:26,474 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:26,474 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,474 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:26,475 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:26,475 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:26,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:26,475 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,476 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:26,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:26,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:26,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:26,477 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:26,479 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:26,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49118 deadline: 1680670646474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,480 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:26,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49112 deadline: 1680670646474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,482 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:26,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49098 deadline: 1680670646476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:26,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49154 deadline: 1680670646475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,491 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:26,491 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:26,492 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:26,492 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:26,492 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:26,498 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:26,498 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:26,498 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:26,498 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:26,498 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:26,498 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:26,498 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:26,498 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:26,498 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,498 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:26,498 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:26,501 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:26,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:26,501 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:26,503 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,503 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:26,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:26,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:26,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:26,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:26,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@496a5189[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:26,505 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:26,505 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,507 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,507 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:26,507 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:26,507 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:26,509 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,509 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:26,510 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,510 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:26,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:26,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:26,510 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:26,510 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:26,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:26,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:26,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:26,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:26,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:26,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:26,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:26,515 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:26,515 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:26,515 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:26,515 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:26,515 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:26,516 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,516 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:26,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,517 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:26,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:26,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:26,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:26,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:26,518 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:26,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:26,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:26,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,520 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:26,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:26,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,522 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:26,522 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:26,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:26,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:26,523 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,523 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:26,523 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:26,523 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,523 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,523 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:26,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:26,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:26,523 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,523 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,523 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:26,523 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:26,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:26,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:26,552 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/b46ea9c5fb4141c6b3c3dcc8684bcfef as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/b46ea9c5fb4141c6b3c3dcc8684bcfef 2023-04-05 04:56:26,559 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ed7bbafaddb97066dca4b03f35c9eee7/C of ed7bbafaddb97066dca4b03f35c9eee7 into b46ea9c5fb4141c6b3c3dcc8684bcfef(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:56:26,559 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:26,560 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7., storeName=ed7bbafaddb97066dca4b03f35c9eee7/C, priority=11, startTime=1680670586031; duration=0sec 2023-04-05 04:56:26,560 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:56:26,562 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=324 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/5bf09ea8b0a544359c430e5a84486fd0 2023-04-05 04:56:26,595 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=324 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/32f830c6270c4a9f884a9d573f476e7c 2023-04-05 04:56:26,623 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:26,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:26,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:26,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:26,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:26,642 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:26,642 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:26,642 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:26,642 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:26,643 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,643 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:26,643 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:26,644 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:26,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:26,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:26,644 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,644 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:26,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:26,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:26,645 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:26,662 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:26,662 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:26,662 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:26,662 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:26,662 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:26,674 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:26,674 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:26,682 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:26,682 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:26,682 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:26,682 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:26,682 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:26,682 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:26,682 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,682 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:26,682 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:26,690 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,690 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,690 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:26,690 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:26,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:26,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:26,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:26,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:26,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:26,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:26,694 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:26,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a2b16ce[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:26,694 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,700 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,700 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:26,700 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:26,700 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:26,701 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:26,701 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:26,701 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:26,702 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,703 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:26,703 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:26,703 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:26,703 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:26,703 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:26,703 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:26,703 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:26,705 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,705 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:26,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,705 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:26,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:26,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:26,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:26,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:26,706 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:26,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:26,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:26,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:26,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:26,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,709 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,713 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:26,713 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:26,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:26,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:26,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:26,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:26,714 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,714 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:26,714 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:26,714 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:26,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:26,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:26,714 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,714 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,714 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,714 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,714 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:26,715 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:26,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:26,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49082 deadline: 1680670646771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,786 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:26,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49112 deadline: 1680670646784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,790 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:26,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49118 deadline: 1680670646788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,792 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:26,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49154 deadline: 1680670646790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,795 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:26,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49098 deadline: 1680670646791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,815 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:26,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:26,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:26,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:26,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:26,819 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:26,819 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:26,819 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:26,819 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:26,819 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,819 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:26,820 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:26,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:26,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:26,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:26,821 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,821 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,821 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:26,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:26,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:26,821 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:26,822 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:26,822 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:26,822 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:26,822 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:26,826 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:26,826 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:26,826 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:26,826 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:26,826 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:26,826 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:26,826 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:26,826 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:26,826 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,826 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:26,827 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:26,843 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,843 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:26,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:26,843 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,843 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:26,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:26,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:26,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:26,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:26,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:26,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@479294dc[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:26,847 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:26,847 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,851 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,852 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:26,852 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:26,852 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:26,852 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,852 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:26,862 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:26,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:26,863 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:26,863 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:26,863 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:26,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:26,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:26,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:26,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:26,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:26,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:26,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:26,869 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:26,869 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:26,869 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:26,869 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:26,869 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:26,870 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,870 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,871 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:26,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:26,871 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:26,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:26,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:26,880 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:26,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:26,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,890 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,890 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:26,890 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:26,890 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,891 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:26,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:26,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:26,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:26,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:26,891 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:26,891 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:26,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:26,891 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:26,891 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,891 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:26,891 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,891 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:26,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:26,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:26,991 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:26,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:26,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:26,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:26,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:26,994 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:26,994 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:26,994 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:26,994 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:26,995 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,995 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:26,996 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:26,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:26,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:26,996 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:26,997 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:26,997 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:26,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:26,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:26,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:26,997 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:26,998 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:26,998 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:26,998 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:26,998 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:26,998 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:27,010 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:27,010 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:27,010 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:27,010 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:27,010 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:27,010 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:27,010 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:27,010 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:27,010 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,010 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:27,010 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:27,012 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,012 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:27,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:27,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:27,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:27,013 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:27,013 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:27,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:27,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:27,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:27,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:27,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48ca702e[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:27,014 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:27,014 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:27,015 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:27,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:27,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:27,015 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:27,015 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:27,015 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:27,016 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,016 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:27,016 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:27,016 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:27,016 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:27,016 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:27,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:27,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:27,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:27,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:27,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:27,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:27,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:27,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:27,026 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:27,026 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:27,026 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:27,026 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:27,026 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:27,030 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:27,030 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:27,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:27,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:27,030 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:27,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:27,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:27,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:27,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:27,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:27,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:27,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:27,032 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:27,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:27,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:27,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,033 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:27,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:27,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:27,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:27,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:27,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:27,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:27,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:27,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:27,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:27,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:27,044 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:27,043 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:27,044 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:27,044 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:27,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:27,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:27,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:27,058 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=324 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/1dfa9e9a00fa479798db87e50d830975 2023-04-05 04:56:27,068 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/5bf09ea8b0a544359c430e5a84486fd0 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/5bf09ea8b0a544359c430e5a84486fd0 2023-04-05 04:56:27,073 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/5bf09ea8b0a544359c430e5a84486fd0, entries=200, sequenceid=324, filesize=14.3 K 2023-04-05 04:56:27,074 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/32f830c6270c4a9f884a9d573f476e7c as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/32f830c6270c4a9f884a9d573f476e7c 2023-04-05 04:56:27,081 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/32f830c6270c4a9f884a9d573f476e7c, entries=150, sequenceid=324, filesize=11.9 K 2023-04-05 04:56:27,082 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/1dfa9e9a00fa479798db87e50d830975 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/1dfa9e9a00fa479798db87e50d830975 2023-04-05 04:56:27,091 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/1dfa9e9a00fa479798db87e50d830975, entries=150, sequenceid=324, filesize=11.9 K 2023-04-05 04:56:27,092 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for ed7bbafaddb97066dca4b03f35c9eee7 in 957ms, sequenceid=324, compaction requested=false 2023-04-05 04:56:27,093 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:27,144 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:27,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:27,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:27,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:27,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:27,155 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:27,155 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:27,155 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:27,155 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:27,156 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:27,156 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:27,156 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,156 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:27,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:27,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:27,157 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,157 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:27,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:27,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:27,158 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:27,160 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:27,160 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:27,160 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:27,160 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:27,160 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:27,174 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:27,174 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:27,174 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:27,174 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:27,174 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ed7bbafaddb97066dca4b03f35c9eee7 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-05 04:56:27,175 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=A 2023-04-05 04:56:27,175 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:27,175 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=B 2023-04-05 04:56:27,175 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:27,175 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=C 2023-04-05 04:56:27,175 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:27,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:27,276 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:27,300 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:27,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49082 deadline: 1680670647297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,302 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:27,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49118 deadline: 1680670647298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,302 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:27,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49112 deadline: 1680670647299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,303 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:27,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49098 deadline: 1680670647299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,303 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:27,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49154 deadline: 1680670647300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:27,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49082 deadline: 1680670647401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,405 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:27,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49118 deadline: 1680670647403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,406 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:27,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49154 deadline: 1680670647406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,407 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:27,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49112 deadline: 1680670647406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,607 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:27,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49082 deadline: 1680670647605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,607 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:27,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49118 deadline: 1680670647606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,611 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:27,611 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:27,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49154 deadline: 1680670647608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49112 deadline: 1680670647609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,619 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/a9dbbf169b7a439585ea201f6f2a47ce 2023-04-05 04:56:27,660 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/bcd4376586c94a3cb87a141ce34ad659 2023-04-05 04:56:27,693 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/c63988d0350741c88af1a98362db12c8 2023-04-05 04:56:27,699 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/a9dbbf169b7a439585ea201f6f2a47ce as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/a9dbbf169b7a439585ea201f6f2a47ce 2023-04-05 04:56:27,705 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/a9dbbf169b7a439585ea201f6f2a47ce, entries=150, sequenceid=340, filesize=11.9 K 2023-04-05 04:56:27,706 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/bcd4376586c94a3cb87a141ce34ad659 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/bcd4376586c94a3cb87a141ce34ad659 2023-04-05 04:56:27,713 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/bcd4376586c94a3cb87a141ce34ad659, entries=150, sequenceid=340, filesize=11.9 K 2023-04-05 04:56:27,714 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/c63988d0350741c88af1a98362db12c8 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/c63988d0350741c88af1a98362db12c8 2023-04-05 04:56:27,723 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/c63988d0350741c88af1a98362db12c8, entries=150, sequenceid=340, filesize=11.9 K 2023-04-05 04:56:27,724 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for ed7bbafaddb97066dca4b03f35c9eee7 in 550ms, sequenceid=340, compaction requested=true 2023-04-05 04:56:27,724 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:27,724 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:27,724 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:27,724 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:27,724 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,724 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:27,725 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:27,726 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:27,726 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:27,726 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:27,727 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:27,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:27,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:27,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:27,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:27,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:27,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:27,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53b786a7[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:27,730 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:27,730 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:27,732 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:27,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:27,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:27,732 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:27,732 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,732 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:27,732 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:27,732 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:27,733 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:27,733 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:27,733 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:27,733 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:27,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:27,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:27,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:27,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:27,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:27,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:27,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:27,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:27,739 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:27,739 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:27,739 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:27,739 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:27,739 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:27,743 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:27,743 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:27,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:27,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:27,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:27,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:27,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:27,743 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:27,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:27,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:27,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:27,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:27,744 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:27,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:27,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:27,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,749 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:27,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:27,751 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,752 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:27,752 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:27,751 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:27,752 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:27,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:27,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:27,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:27,752 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:27,752 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,752 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:27,752 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:27,752 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:27,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:27,752 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:27,752 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:27,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:27,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:27,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:27,852 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:27,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:27,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:27,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:27,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:27,855 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:27,855 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:27,855 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:27,855 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:27,856 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:27,856 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:27,857 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:27,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:27,857 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:27,857 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,857 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:27,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:27,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:27,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:27,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:27,858 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:27,858 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:27,858 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:27,858 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:27,858 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:27,858 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:27,858 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:27,858 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:27,858 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:27,858 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:27,858 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ed7bbafaddb97066dca4b03f35c9eee7 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-05 04:56:27,859 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=A 2023-04-05 04:56:27,859 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:27,859 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=B 2023-04-05 04:56:27,859 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:27,859 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=C 2023-04-05 04:56:27,859 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:27,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:27,911 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:27,925 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:27,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49118 deadline: 1680670647923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,925 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:27,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49112 deadline: 1680670647924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,927 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:27,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49154 deadline: 1680670647924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:27,929 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:27,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49082 deadline: 1680670647925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:28,027 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:28,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49118 deadline: 1680670648026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:28,027 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:28,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49112 deadline: 1680670648026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:28,031 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:28,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49154 deadline: 1680670648028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:28,031 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:28,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49082 deadline: 1680670648030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:28,230 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:28,230 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:28,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49118 deadline: 1680670648229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:28,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49112 deadline: 1680670648229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:28,234 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:28,234 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:28,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49154 deadline: 1680670648232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:28,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49082 deadline: 1680670648232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:28,268 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/1a120935c3904216bd751d558c07d002 2023-04-05 04:56:28,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:28,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49098 deadline: 1680670648311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:28,532 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:28,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49112 deadline: 1680670648532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:28,535 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:28,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49118 deadline: 1680670648533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:28,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:28,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49082 deadline: 1680670648537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:28,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:28,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49154 deadline: 1680670648537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:28,689 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/b1941cae3d7b4392ac24e48819576222 2023-04-05 04:56:29,037 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:29,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49118 deadline: 1680670649036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:29,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49112 deadline: 1680670649037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,043 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:29,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49154 deadline: 1680670649043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,046 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:29,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49082 deadline: 1680670649045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,113 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/a9f332e8e2dc454d94c99c05e0a3af83 2023-04-05 04:56:29,120 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/1a120935c3904216bd751d558c07d002 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/1a120935c3904216bd751d558c07d002 2023-04-05 04:56:29,126 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/1a120935c3904216bd751d558c07d002, entries=150, sequenceid=357, filesize=11.9 K 2023-04-05 04:56:29,127 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/b1941cae3d7b4392ac24e48819576222 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/b1941cae3d7b4392ac24e48819576222 2023-04-05 04:56:29,132 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/b1941cae3d7b4392ac24e48819576222, entries=150, sequenceid=357, filesize=11.9 K 2023-04-05 04:56:29,133 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/a9f332e8e2dc454d94c99c05e0a3af83 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/a9f332e8e2dc454d94c99c05e0a3af83 2023-04-05 04:56:29,149 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/a9f332e8e2dc454d94c99c05e0a3af83, entries=150, sequenceid=357, filesize=11.9 K 2023-04-05 04:56:29,152 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for ed7bbafaddb97066dca4b03f35c9eee7 in 1294ms, sequenceid=357, compaction requested=true 2023-04-05 04:56:29,152 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:29,152 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:29,152 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:29,152 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:29,152 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,152 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:29,152 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:29,153 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,153 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:29,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:29,153 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,153 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:29,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:29,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:29,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:29,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:29,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:29,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21d86779[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:29,155 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:29,155 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,158 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,158 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:29,158 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:29,158 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:29,158 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,158 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:29,159 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,159 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:29,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:29,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:29,159 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:29,159 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:29,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:29,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:29,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:29,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:29,164 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:29,164 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:29,164 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:29,164 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:29,164 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:29,168 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,168 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:29,168 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:29,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:29,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:29,169 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:29,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:29,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:29,172 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:29,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:29,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,179 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,179 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:29,179 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:29,179 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:29,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:29,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:29,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:29,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:29,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:29,179 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:29,179 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,180 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,180 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,181 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:29,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:29,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:29,181 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:29,181 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:29,281 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:29,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:29,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:29,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:29,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:29,289 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:29,290 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:29,290 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:29,290 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:29,291 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,291 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:29,291 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:29,291 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:29,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:29,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:29,292 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,292 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:29,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:29,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:29,292 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:29,293 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:29,293 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:29,293 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:29,293 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:29,293 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:29,293 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:29,293 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:29,293 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:29,293 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:29,293 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ed7bbafaddb97066dca4b03f35c9eee7 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-05 04:56:29,293 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=A 2023-04-05 04:56:29,294 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:29,294 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=B 2023-04-05 04:56:29,294 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:29,294 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=C 2023-04-05 04:56:29,294 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:29,312 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/4c9f904899c34d79ae20005ac5309f57 2023-04-05 04:56:29,328 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/8202fbbd4d304d8e8246d19d967b4bd0 2023-04-05 04:56:29,340 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/ef3215a69a6942dfad75f6f4647a5a4b 2023-04-05 04:56:29,346 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/4c9f904899c34d79ae20005ac5309f57 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/4c9f904899c34d79ae20005ac5309f57 2023-04-05 04:56:29,354 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/4c9f904899c34d79ae20005ac5309f57, entries=150, sequenceid=372, filesize=11.9 K 2023-04-05 04:56:29,358 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/8202fbbd4d304d8e8246d19d967b4bd0 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/8202fbbd4d304d8e8246d19d967b4bd0 2023-04-05 04:56:29,365 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/8202fbbd4d304d8e8246d19d967b4bd0, entries=150, sequenceid=372, filesize=11.9 K 2023-04-05 04:56:29,367 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/ef3215a69a6942dfad75f6f4647a5a4b as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/ef3215a69a6942dfad75f6f4647a5a4b 2023-04-05 04:56:29,374 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/ef3215a69a6942dfad75f6f4647a5a4b, entries=150, sequenceid=372, filesize=11.9 K 2023-04-05 04:56:29,375 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for ed7bbafaddb97066dca4b03f35c9eee7 in 82ms, sequenceid=372, compaction requested=true 2023-04-05 04:56:29,375 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:29,375 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:29,375 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:29,375 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:29,375 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,375 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:29,375 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:29,376 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,376 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:29,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:29,376 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,377 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:29,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:29,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:29,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:29,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:29,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:29,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@723f23ea[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:29,378 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:29,378 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,379 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,379 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:29,379 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:29,379 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:29,380 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,380 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:29,380 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:29,380 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,380 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:29,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:29,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:29,380 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:29,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:29,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:29,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:29,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:29,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:29,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:29,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:29,385 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:29,385 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:29,385 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:29,385 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:29,385 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:29,387 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,387 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,387 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:29,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:29,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:29,388 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:29,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:29,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:29,389 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:29,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:29,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,391 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,391 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:29,391 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:29,391 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,391 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,391 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:29,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:29,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:29,391 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:29,391 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,391 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:29,391 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:29,391 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:29,392 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:29,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:29,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:29,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:29,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:29,492 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:29,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:29,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:29,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:29,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:29,495 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:29,495 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:29,495 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:29,495 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:29,495 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,495 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:29,496 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:29,496 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:29,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:29,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:29,497 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,497 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:29,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:29,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:29,498 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:29,498 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:29,498 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:29,498 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:29,498 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:29,498 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:29,499 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:29,499 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:29,499 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:29,499 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:29,499 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:29,499 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:29,499 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:29,499 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:29,499 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,499 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:29,499 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:29,500 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:29,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:29,501 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:29,501 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,501 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:29,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:29,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:29,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:29,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:29,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@696f7683[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:29,502 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:29,503 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,503 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,504 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:29,504 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:29,504 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,504 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:29,504 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:29,506 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,506 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:29,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:29,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:29,506 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:29,506 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:29,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:29,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:29,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:29,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:29,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:29,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:29,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:29,508 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:29,508 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:29,508 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:29,509 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:29,509 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:29,510 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,510 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:29,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,510 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:29,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:29,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:29,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:29,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,511 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:29,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:29,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:29,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:29,512 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:29,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:29,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,514 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:29,514 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:29,514 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,514 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:29,515 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:29,515 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,515 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,515 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,515 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,515 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:29,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:29,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:29,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:29,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:29,515 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:29,515 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:29,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:29,615 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:29,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:29,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:29,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:29,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:29,618 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:29,618 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:29,618 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:29,618 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:29,619 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,619 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:29,619 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:29,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:29,619 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:29,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:29,620 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,620 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:29,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:29,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:29,621 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:29,621 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:29,622 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:29,622 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:29,622 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:29,622 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:29,626 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:29,626 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:29,626 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:29,626 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:29,626 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:29,626 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:29,626 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:29,626 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:29,626 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,626 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:29,626 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:29,635 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:29,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:29,635 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,635 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,635 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:29,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:29,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:29,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:29,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:29,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:29,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68257b7[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:29,637 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:29,637 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,640 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,640 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:29,640 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:29,640 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:29,641 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,641 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:29,642 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:29,642 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:29,642 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:29,642 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:29,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:29,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:29,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:29,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:29,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:29,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:29,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:29,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:29,648 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:29,648 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:29,648 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:29,648 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:29,648 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:29,649 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,649 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:29,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,649 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:29,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:29,650 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:29,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:29,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:29,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,652 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:29,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:29,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,660 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,660 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:29,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:29,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:29,660 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:29,660 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,660 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,660 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,660 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,660 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:29,660 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:29,660 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:29,661 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:29,661 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:29,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:29,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:29,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:29,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:29,761 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:29,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:29,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:29,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:29,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:29,765 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:29,765 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:29,765 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:29,765 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:29,767 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,767 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:29,768 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:29,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:29,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:29,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:29,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:29,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:29,769 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:29,769 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:29,769 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:29,769 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:29,769 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:29,769 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:29,771 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,771 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,771 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:29,774 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:29,774 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:29,774 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:29,774 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:29,774 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:29,774 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:29,774 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:29,774 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:29,774 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,774 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:29,774 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:29,779 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,779 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,779 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:29,780 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:29,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:29,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:29,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:29,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:29,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:29,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:29,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64830bb1[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:29,790 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:29,790 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,791 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,791 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:29,791 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,791 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:29,791 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:29,791 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:29,792 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:29,792 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:29,792 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:29,792 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:29,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:29,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:29,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:29,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:29,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:29,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:29,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:29,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:29,795 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:29,795 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:29,795 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:29,795 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:29,795 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:29,804 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,804 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:29,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:29,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,804 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:29,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:29,805 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:29,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:29,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:29,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:29,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:29,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,807 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:29,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:29,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,812 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,812 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:29,812 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:29,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:29,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:29,812 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:29,812 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:29,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:29,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:29,812 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:29,813 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:29,812 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:29,812 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:29,813 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,813 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,813 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:29,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:29,913 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:29,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:29,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:29,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:29,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:29,918 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:29,919 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:29,919 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:29,919 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:29,919 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,919 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:29,922 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:29,922 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:29,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:29,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:29,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,923 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,923 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:29,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:29,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:29,923 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:29,924 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:29,924 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:29,924 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:29,924 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:29,926 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:29,926 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:29,926 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:29,926 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:29,926 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:29,926 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:29,926 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:29,926 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:29,926 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,926 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:29,926 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:29,935 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,935 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:29,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:29,935 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:29,935 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:29,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:29,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:29,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:29,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:29,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fa854b8[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:29,938 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:29,938 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,938 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,939 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:29,939 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:29,939 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:29,939 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,939 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:29,940 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,940 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:29,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:29,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:29,940 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:29,940 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:29,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:29,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:29,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:29,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:29,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:29,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:29,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:29,942 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:29,942 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:29,942 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:29,942 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:29,942 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:29,943 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,943 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:29,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:29,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:29,944 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:29,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:29,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:29,945 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:29,946 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:29,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:29,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:29,955 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:29,955 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:29,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:29,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:29,955 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,955 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:29,955 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:29,955 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:29,955 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,955 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:29,955 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:29,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:29,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:29,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:29,955 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:29,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:29,955 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:29,956 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:29,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:30,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:30,054 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ed7bbafaddb97066dca4b03f35c9eee7 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 04:56:30,055 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=A 2023-04-05 04:56:30,055 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:30,055 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=B 2023-04-05 04:56:30,055 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:30,056 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=C 2023-04-05 04:56:30,056 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:30,056 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:30,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:30,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:30,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:30,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:30,066 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:30,066 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:30,066 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:30,066 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:30,066 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,066 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:30,067 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,067 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,067 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:30,068 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:30,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:30,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:30,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:30,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:30,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:30,069 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:30,070 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:30,070 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:30,070 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:30,070 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:30,070 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:30,070 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:30,070 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:30,070 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:30,070 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:30,070 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:30,070 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:30,070 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:30,070 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/edd32493d15a468185f74f510d33f537 2023-04-05 04:56:30,070 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:30,071 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,071 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:30,071 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:30,085 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:30,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49112 deadline: 1680670650081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:30,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49082 deadline: 1680670650082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,087 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670590072, max timestamp=1680670590082Num uniques 859; 2023-04-05 04:56:30,087 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670590072, max timestamp=1680670590082Num uniques 859; 2023-04-05 04:56:30,087 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:56:30,087 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:30,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:30,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680670590072, max timestamp=1680670590082Num uniques 859; 2023-04-05 04:56:30,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-05 04:56:30,088 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/aa80bd06b5934f94a28df180dce756d4 2023-04-05 04:56:30,087 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:56:30,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:30,087 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,089 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,089 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:30,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:30,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:30,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:30,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7de1eca0[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:30,090 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:30,090 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,093 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,093 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:30,093 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:30,093 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:30,094 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,094 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:30,094 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:30,094 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,094 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:30,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:30,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:30,095 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:30,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:30,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:30,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:30,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:30,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:30,099 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:30,099 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:30,099 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:30,099 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:30,099 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:30,101 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,101 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:30,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:30,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,101 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:30,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:30,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:30,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:30,102 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:30,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,104 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:30,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,111 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/ce159ded61874157940369cb59467671 2023-04-05 04:56:30,114 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,114 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:30,114 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:30,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:30,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:30,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:30,115 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:30,115 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:30,114 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:30,116 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,116 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,116 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,116 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,114 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:30,118 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:30,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:30,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:30,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:30,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:30,119 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/edd32493d15a468185f74f510d33f537 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/edd32493d15a468185f74f510d33f537 2023-04-05 04:56:30,132 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/edd32493d15a468185f74f510d33f537, entries=300, sequenceid=387, filesize=19.0 K 2023-04-05 04:56:30,134 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/aa80bd06b5934f94a28df180dce756d4 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/aa80bd06b5934f94a28df180dce756d4 2023-04-05 04:56:30,138 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/aa80bd06b5934f94a28df180dce756d4, entries=150, sequenceid=387, filesize=11.9 K 2023-04-05 04:56:30,140 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/ce159ded61874157940369cb59467671 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/ce159ded61874157940369cb59467671 2023-04-05 04:56:30,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:30,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49154 deadline: 1680670650133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,151 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/ce159ded61874157940369cb59467671, entries=150, sequenceid=387, filesize=11.9 K 2023-04-05 04:56:30,152 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=124.84 KB/127835 for ed7bbafaddb97066dca4b03f35c9eee7 in 98ms, sequenceid=387, compaction requested=true 2023-04-05 04:56:30,152 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:30,152 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:56:30,152 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-05 04:56:30,152 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:56:30,152 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-05 04:56:30,153 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-05 04:56:30,158 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 83626 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-05 04:56:30,158 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): ed7bbafaddb97066dca4b03f35c9eee7/A is initiating minor compaction (all files) 2023-04-05 04:56:30,158 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ed7bbafaddb97066dca4b03f35c9eee7/A in TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:30,159 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/e464dfa7666943379ae0b788e3315c54, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/5bf09ea8b0a544359c430e5a84486fd0, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/a9dbbf169b7a439585ea201f6f2a47ce, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/1a120935c3904216bd751d558c07d002, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/4c9f904899c34d79ae20005ac5309f57, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/edd32493d15a468185f74f510d33f537] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp, totalSize=81.7 K 2023-04-05 04:56:30,159 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73866 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-05 04:56:30,159 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): ed7bbafaddb97066dca4b03f35c9eee7/B is initiating minor compaction (all files) 2023-04-05 04:56:30,159 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ed7bbafaddb97066dca4b03f35c9eee7/B in TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:30,159 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/d04cc117816545fea7621d1ab431aa53, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/32f830c6270c4a9f884a9d573f476e7c, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/bcd4376586c94a3cb87a141ce34ad659, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/b1941cae3d7b4392ac24e48819576222, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/8202fbbd4d304d8e8246d19d967b4bd0, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/aa80bd06b5934f94a28df180dce756d4] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp, totalSize=72.1 K 2023-04-05 04:56:30,160 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting e464dfa7666943379ae0b788e3315c54, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1680670585370 2023-04-05 04:56:30,162 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting d04cc117816545fea7621d1ab431aa53, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1680670585370 2023-04-05 04:56:30,162 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 5bf09ea8b0a544359c430e5a84486fd0, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=324, earliestPutTs=1680670585518 2023-04-05 04:56:30,162 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 32f830c6270c4a9f884a9d573f476e7c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=324, earliestPutTs=1680670585518 2023-04-05 04:56:30,163 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting a9dbbf169b7a439585ea201f6f2a47ce, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1680670586158 2023-04-05 04:56:30,163 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting bcd4376586c94a3cb87a141ce34ad659, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1680670586158 2023-04-05 04:56:30,163 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting b1941cae3d7b4392ac24e48819576222, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1680670587296 2023-04-05 04:56:30,164 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 1a120935c3904216bd751d558c07d002, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1680670587296 2023-04-05 04:56:30,165 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 8202fbbd4d304d8e8246d19d967b4bd0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1680670587922 2023-04-05 04:56:30,166 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting aa80bd06b5934f94a28df180dce756d4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1680670590050 2023-04-05 04:56:30,167 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 4c9f904899c34d79ae20005ac5309f57, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1680670587922 2023-04-05 04:56:30,167 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting edd32493d15a468185f74f510d33f537, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1680670590047 2023-04-05 04:56:30,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:30,200 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ed7bbafaddb97066dca4b03f35c9eee7 3/3 column families, dataSize=138.26 KB heapSize=394.23 KB 2023-04-05 04:56:30,200 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=A 2023-04-05 04:56:30,201 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-05 04:56:30,201 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=B 2023-04-05 04:56:30,201 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-05 04:56:30,201 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=C 2023-04-05 04:56:30,201 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-05 04:56:30,210 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): ed7bbafaddb97066dca4b03f35c9eee7#A#compaction#83 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:56:30,215 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:30,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:30,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:30,228 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): ed7bbafaddb97066dca4b03f35c9eee7#B#compaction#85 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:56:30,242 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:30,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49082 deadline: 1680670650231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,248 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:30,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49118 deadline: 1680670650236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:30,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:30,248 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:30,248 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:30,248 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:30,248 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:30,249 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,249 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:30,251 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:30,251 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:30,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:30,251 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,251 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:30,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:30,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:30,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:30,252 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:30,254 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:30,254 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:30,254 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:30,254 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:30,254 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:30,260 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:30,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49154 deadline: 1680670650242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,261 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:30,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49112 deadline: 1680670650242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,261 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:30,261 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:30,261 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:30,261 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:30,261 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:30,261 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:30,261 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:30,261 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:30,261 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,266 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:30,266 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:30,269 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,269 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:30,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:30,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:30,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:30,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:30,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:30,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@694a2344[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:30,275 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:30,275 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,278 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,278 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:30,280 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,281 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:30,281 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:30,281 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:30,282 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,282 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:30,286 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,286 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:30,286 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:30,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,286 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/4ff3f16da04144d3926ca18fb79dea4f as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/4ff3f16da04144d3926ca18fb79dea4f 2023-04-05 04:56:30,286 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:30,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:30,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:30,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:30,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:30,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:30,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:30,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:30,293 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:30,293 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:30,293 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:30,294 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:30,294 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:30,294 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in ed7bbafaddb97066dca4b03f35c9eee7/B of ed7bbafaddb97066dca4b03f35c9eee7 into 4ff3f16da04144d3926ca18fb79dea4f(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:56:30,294 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:30,294 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7., storeName=ed7bbafaddb97066dca4b03f35c9eee7/B, priority=10, startTime=1680670590152; duration=0sec 2023-04-05 04:56:30,294 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:56:30,294 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-05 04:56:30,295 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,295 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:30,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,296 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:30,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:30,296 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:30,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:30,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:30,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,297 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:30,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,298 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73832 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-05 04:56:30,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,299 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): ed7bbafaddb97066dca4b03f35c9eee7/C is initiating minor compaction (all files) 2023-04-05 04:56:30,299 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ed7bbafaddb97066dca4b03f35c9eee7/C in TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:30,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:30,299 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/b46ea9c5fb4141c6b3c3dcc8684bcfef, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/1dfa9e9a00fa479798db87e50d830975, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/c63988d0350741c88af1a98362db12c8, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/a9f332e8e2dc454d94c99c05e0a3af83, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/ef3215a69a6942dfad75f6f4647a5a4b, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/ce159ded61874157940369cb59467671] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp, totalSize=72.1 K 2023-04-05 04:56:30,299 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,299 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:30,299 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:30,300 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:30,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:30,300 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,300 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,300 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:30,300 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,300 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting b46ea9c5fb4141c6b3c3dcc8684bcfef, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1680670585370 2023-04-05 04:56:30,300 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:30,300 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:30,300 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:30,300 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:30,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:30,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:30,300 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 1dfa9e9a00fa479798db87e50d830975, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=324, earliestPutTs=1680670585518 2023-04-05 04:56:30,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:30,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:30,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:30,301 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting c63988d0350741c88af1a98362db12c8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1680670586158 2023-04-05 04:56:30,301 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting a9f332e8e2dc454d94c99c05e0a3af83, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1680670587296 2023-04-05 04:56:30,301 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting ef3215a69a6942dfad75f6f4647a5a4b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1680670587922 2023-04-05 04:56:30,302 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting ce159ded61874157940369cb59467671, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1680670590050 2023-04-05 04:56:30,328 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): ed7bbafaddb97066dca4b03f35c9eee7#C#compaction#86 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:56:30,335 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:30,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49098 deadline: 1680670650334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,349 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4178 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7., hostname=jenkins-hbase20.apache.org,38091,1680670569206, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 04:56:30,351 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:30,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49082 deadline: 1680670650344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,352 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:30,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49118 deadline: 1680670650351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,363 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:30,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49112 deadline: 1680670650363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,400 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:30,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:30,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:30,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:30,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:30,406 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:30,406 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:30,406 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:30,406 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:30,407 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,407 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:30,409 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:30,409 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:30,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:30,409 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,409 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:30,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:30,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:30,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:30,410 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:30,410 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:30,410 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:30,410 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:30,410 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:30,411 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:30,414 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:30,414 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:30,414 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:30,414 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:30,414 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:30,414 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:30,414 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:30,414 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:30,414 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,415 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:30,415 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:30,417 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,417 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:30,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:30,417 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,417 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:30,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:30,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:30,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:30,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:30,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15e8735f[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:30,419 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:30,419 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,420 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,420 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:30,420 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:30,420 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,420 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:30,420 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:30,422 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,422 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:30,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:30,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:30,422 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:30,422 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:30,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:30,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:30,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:30,424 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:30,424 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:30,424 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:30,424 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:30,424 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:30,425 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,425 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:30,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:30,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,425 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:30,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:30,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:30,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:30,425 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:30,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,426 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:30,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,429 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:30,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:30,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:30,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:30,429 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,429 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:30,429 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:30,429 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,430 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:30,430 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:30,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:30,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:30,430 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:30,430 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,430 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:30,430 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,430 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:30,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:30,466 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:30,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49154 deadline: 1680670650463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,530 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:30,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:30,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:30,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:30,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:30,533 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:30,533 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:30,533 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:30,533 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:30,533 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,533 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:30,534 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:30,534 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:30,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:30,534 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,534 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:30,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:30,534 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:30,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:30,535 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:30,536 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:30,536 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:30,536 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:30,536 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:30,536 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:30,536 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:30,536 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:30,536 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:30,536 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:30,536 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:30,536 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:30,536 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:30,536 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:30,536 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,536 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:30,536 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:30,537 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,537 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:30,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:30,538 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,538 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:30,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:30,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:30,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:30,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:30,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38c8e8ce[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:30,539 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:30,539 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,540 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,540 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:30,540 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:30,540 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,540 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:30,540 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:30,541 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,541 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:30,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:30,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:30,541 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:30,541 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:30,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:30,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:30,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:30,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:30,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:30,544 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:30,544 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:30,544 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:30,544 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:30,544 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:30,545 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,545 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,545 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:30,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:30,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:30,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:30,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:30,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:30,546 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:30,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,548 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:30,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,550 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,550 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:30,550 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:30,550 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:30,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:30,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:30,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:30,550 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:30,550 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,550 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,550 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,550 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:30,550 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:30,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:30,550 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:30,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:30,550 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:30,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:30,555 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:30,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49118 deadline: 1680670650553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,556 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:30,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49082 deadline: 1680670650554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,569 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:30,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49112 deadline: 1680670650565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,650 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:30,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:30,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:30,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:30,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:30,653 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:30,653 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:30,653 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:30,653 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:30,654 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,654 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:30,654 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:30,654 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:30,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:30,655 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,655 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:30,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:30,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:30,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:30,655 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:30,656 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:30,656 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:30,656 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:30,656 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:30,656 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:30,656 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:30,656 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:30,656 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:30,656 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:30,656 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:30,656 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:30,656 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:30,656 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:30,656 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,656 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:30,656 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:30,657 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=425 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/1e123758893f4a8cb7d199af07f03db3 2023-04-05 04:56:30,658 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,658 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:30,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:30,658 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,658 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:30,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:30,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:30,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:30,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:30,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4abe879b[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:30,661 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:30,661 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,661 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,662 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:30,662 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:30,662 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:30,662 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,663 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:30,663 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,664 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:30,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:30,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:30,664 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:30,664 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:30,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:30,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:30,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:30,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:30,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:30,666 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:30,667 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:30,667 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:30,667 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:30,667 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:30,668 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,668 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,668 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:30,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:30,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:30,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,669 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/739d412b65b5411bb6674a05d93e508e as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/739d412b65b5411bb6674a05d93e508e 2023-04-05 04:56:30,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:30,669 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:30,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:30,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:30,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,671 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:30,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,674 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:30,674 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:30,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:30,674 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:30,674 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:30,674 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,674 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,674 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,674 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,674 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:30,674 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:30,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:30,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:30,675 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:30,675 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:30,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:30,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:30,676 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in ed7bbafaddb97066dca4b03f35c9eee7/A of ed7bbafaddb97066dca4b03f35c9eee7 into 739d412b65b5411bb6674a05d93e508e(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:56:30,676 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:30,676 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7., storeName=ed7bbafaddb97066dca4b03f35c9eee7/A, priority=10, startTime=1680670590152; duration=0sec 2023-04-05 04:56:30,676 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:56:30,775 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:30,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:30,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:30,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:30,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49154 deadline: 1680670650773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,780 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/80428c7c04fc47008f7b50893a413166 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/80428c7c04fc47008f7b50893a413166 2023-04-05 04:56:30,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:30,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:30,787 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:30,787 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:30,787 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:30,787 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:30,787 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in ed7bbafaddb97066dca4b03f35c9eee7/C of ed7bbafaddb97066dca4b03f35c9eee7 into 80428c7c04fc47008f7b50893a413166(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:56:30,787 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:30,787 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7., storeName=ed7bbafaddb97066dca4b03f35c9eee7/C, priority=10, startTime=1680670590152; duration=0sec 2023-04-05 04:56:30,787 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:56:30,787 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,787 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:30,788 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:30,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:30,788 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:30,789 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,789 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:30,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:30,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:30,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:30,789 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:30,790 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:30,790 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:30,790 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:30,790 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:30,790 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:30,790 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:30,790 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:30,790 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:30,790 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:30,790 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:30,790 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:30,790 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:30,790 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:30,790 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,790 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:30,791 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:30,791 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,792 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:30,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:30,792 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,792 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:30,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:30,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:30,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:30,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:30,793 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:30,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72d6cd8b[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:30,793 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,794 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,794 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:30,794 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:30,794 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:30,794 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,794 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:30,795 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:30,795 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,795 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:30,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:30,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:30,795 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:30,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:30,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:30,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:30,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:30,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:30,799 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:30,799 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:30,799 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:30,799 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:30,799 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:30,804 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,804 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:30,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,804 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:30,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:30,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:30,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:30,807 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:30,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:30,809 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:30,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,812 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,812 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:30,812 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:30,812 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:30,812 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:30,812 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:30,812 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:30,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:30,812 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,812 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:30,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:30,813 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:30,813 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:30,812 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:30,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:30,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:30,857 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:30,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49082 deadline: 1680670650857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,860 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:30,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49118 deadline: 1680670650858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,871 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:30,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49112 deadline: 1680670650870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,913 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:30,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:30,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:30,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:30,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:30,916 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:30,916 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:30,916 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:30,916 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:30,916 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,916 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:30,917 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:30,917 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:30,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:30,918 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,918 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:30,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:30,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:30,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:30,918 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:30,919 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:30,919 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:30,919 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:30,919 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:30,919 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:30,919 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:30,919 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:30,919 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:30,919 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:30,919 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:30,919 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:30,919 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:30,919 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:30,919 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,919 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:30,919 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:30,920 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,920 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:30,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:30,920 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,921 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:30,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:30,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:30,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:30,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:30,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24f22a05[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:30,922 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:30,922 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,922 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,923 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:30,923 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:30,923 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:30,923 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,923 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:30,923 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:30,923 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:30,923 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,923 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:30,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:30,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:30,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:30,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:30,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:30,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:30,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:30,926 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:30,926 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:30,926 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:30,926 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:30,926 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:30,927 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,927 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,927 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:30,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:30,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:30,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:30,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:30,927 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:30,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:30,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,928 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:30,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:30,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,930 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,930 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:30,930 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:30,930 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:30,930 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:30,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:30,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:30,930 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:30,930 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,930 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:30,930 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:30,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:30,930 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:30,930 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:30,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:30,930 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:30,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:30,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:30,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:31,030 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:31,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:31,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:31,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:31,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:31,035 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:31,035 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:31,035 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:31,035 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:31,035 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,035 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:31,036 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:31,036 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:31,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:31,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:31,037 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,037 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:31,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:31,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:31,038 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:31,038 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:31,038 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:31,038 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:31,038 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:31,038 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:31,039 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:31,039 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:31,039 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:31,039 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:31,039 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:31,039 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:31,039 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:31,039 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:31,039 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,039 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:31,039 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:31,040 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,040 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:31,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:31,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:31,041 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,041 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:31,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:31,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:31,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:31,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:31,043 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:31,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b0328cd[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:31,043 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,044 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,044 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,044 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:31,044 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:31,044 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:31,044 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:31,045 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,045 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:31,045 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:31,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:31,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:31,045 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:31,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:31,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:31,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:31,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:31,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:31,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:31,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:31,048 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:31,048 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:31,048 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:31,048 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:31,048 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:31,051 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,051 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:31,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:31,051 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:31,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:31,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:31,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:31,053 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:31,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:31,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:31,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,055 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:31,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:31,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,060 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,060 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:31,060 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:31,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:31,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:31,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:31,060 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:31,060 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,060 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,060 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,060 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,060 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:31,060 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:31,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:31,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:31,060 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:31,060 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:31,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:31,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:31,084 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.57 KB at sequenceid=425 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/9abbd6a67c1f482e989615e0a8c0ea74 2023-04-05 04:56:31,112 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=425 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/c15659ac25af48029454ff74506a2106 2023-04-05 04:56:31,118 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/1e123758893f4a8cb7d199af07f03db3 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/1e123758893f4a8cb7d199af07f03db3 2023-04-05 04:56:31,126 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/1e123758893f4a8cb7d199af07f03db3, entries=300, sequenceid=425, filesize=19.0 K 2023-04-05 04:56:31,127 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/9abbd6a67c1f482e989615e0a8c0ea74 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/9abbd6a67c1f482e989615e0a8c0ea74 2023-04-05 04:56:31,140 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/9abbd6a67c1f482e989615e0a8c0ea74, entries=150, sequenceid=425, filesize=11.9 K 2023-04-05 04:56:31,142 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/c15659ac25af48029454ff74506a2106 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/c15659ac25af48029454ff74506a2106 2023-04-05 04:56:31,153 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/c15659ac25af48029454ff74506a2106, entries=150, sequenceid=425, filesize=11.9 K 2023-04-05 04:56:31,156 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~144.97 KB/148445, heapSize ~414.80 KB/424760, currentSize=40.25 KB/41220 for ed7bbafaddb97066dca4b03f35c9eee7 in 957ms, sequenceid=425, compaction requested=false 2023-04-05 04:56:31,156 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:31,160 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:31,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:31,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:31,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:31,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:31,175 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:31,175 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:31,175 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:31,175 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:31,175 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,175 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:31,177 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:31,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:31,177 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:31,177 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,177 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:31,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:31,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:31,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:31,178 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:31,180 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:31,180 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:31,180 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:31,180 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:31,180 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:31,182 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:31,182 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:31,182 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:31,182 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:31,183 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ed7bbafaddb97066dca4b03f35c9eee7 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-05 04:56:31,184 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=A 2023-04-05 04:56:31,184 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:31,186 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=B 2023-04-05 04:56:31,186 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:31,186 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=C 2023-04-05 04:56:31,186 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:31,225 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=437 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/f5012106f8e54e3ea9b66c161428bf17 2023-04-05 04:56:31,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:31,283 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:31,291 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=437 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/08163ef5a2364ba1ae4c3c50fa871d6f 2023-04-05 04:56:31,334 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=437 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/880d44ffbc08487680a579a1f8d91419 2023-04-05 04:56:31,341 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/f5012106f8e54e3ea9b66c161428bf17 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/f5012106f8e54e3ea9b66c161428bf17 2023-04-05 04:56:31,358 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/f5012106f8e54e3ea9b66c161428bf17, entries=150, sequenceid=437, filesize=11.9 K 2023-04-05 04:56:31,360 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/08163ef5a2364ba1ae4c3c50fa871d6f as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/08163ef5a2364ba1ae4c3c50fa871d6f 2023-04-05 04:56:31,372 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/08163ef5a2364ba1ae4c3c50fa871d6f, entries=150, sequenceid=437, filesize=11.9 K 2023-04-05 04:56:31,374 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/880d44ffbc08487680a579a1f8d91419 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/880d44ffbc08487680a579a1f8d91419 2023-04-05 04:56:31,381 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/880d44ffbc08487680a579a1f8d91419, entries=150, sequenceid=437, filesize=11.9 K 2023-04-05 04:56:31,383 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=114.05 KB/116790 for ed7bbafaddb97066dca4b03f35c9eee7 in 201ms, sequenceid=437, compaction requested=true 2023-04-05 04:56:31,383 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:31,383 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:31,383 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:31,383 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:31,383 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,383 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:31,383 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:31,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:31,384 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ed7bbafaddb97066dca4b03f35c9eee7 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-05 04:56:31,384 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:31,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:31,384 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=A 2023-04-05 04:56:31,385 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:56:31,385 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=B 2023-04-05 04:56:31,385 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:56:31,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:31,384 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,385 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=C 2023-04-05 04:56:31,385 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:56:31,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:31,385 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,385 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:31,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:31,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:31,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:31,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65bfee69[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:31,387 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:31,387 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,388 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,388 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:31,388 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:31,388 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:31,388 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,388 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:31,389 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,389 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:31,389 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:31,389 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:31,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:31,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:31,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:31,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:31,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:31,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:31,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:31,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:31,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:31,394 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:31,394 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:31,394 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:31,394 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:31,394 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:31,396 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,396 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,396 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:31,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:31,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:31,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:31,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:31,397 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:31,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:31,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:31,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:31,402 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:31,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:31,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,410 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:31,410 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:31,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:31,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:31,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:31,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:31,410 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:31,410 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,410 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:31,410 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:31,410 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,410 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,410 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,410 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,410 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:31,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:31,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:31,410 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:31,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:31,436 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:31,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49082 deadline: 1680670651404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,440 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:31,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49118 deadline: 1680670651436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,441 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:31,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49154 deadline: 1680670651436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,446 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:31,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49112 deadline: 1680670651441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,511 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:31,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:31,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:31,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:31,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:31,513 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:31,513 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:31,514 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:31,514 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:31,514 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,514 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:31,515 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,515 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:31,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:31,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:31,515 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,515 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:31,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:31,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:31,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:31,515 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:31,516 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:31,516 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:31,516 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:31,516 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:31,516 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:31,518 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:31,518 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:31,518 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:31,518 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:31,518 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:31,518 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:31,518 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:31,518 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:31,518 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,518 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:31,518 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:31,520 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:31,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:31,521 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:31,521 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,521 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:31,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:31,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:31,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:31,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:31,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2141ddc0[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:31,523 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:31,523 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,525 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,525 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:31,525 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,525 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:31,525 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:31,525 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:31,526 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,526 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:31,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:31,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:31,526 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:31,526 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:31,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:31,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:31,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:31,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:31,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:31,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:31,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:31,530 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:31,530 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:31,530 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:31,530 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:31,530 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:31,532 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,533 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,533 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:31,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,533 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:31,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:31,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:31,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:31,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:31,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:31,534 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,536 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,536 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:31,536 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:31,536 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:31,536 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,536 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,537 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:31,537 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:31,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:31,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:31,537 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:31,537 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:31,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:31,537 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:31,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:31,538 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:31,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:31,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:31,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:31,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49118 deadline: 1680670651541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,544 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:31,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49082 deadline: 1680670651542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,544 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:31,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49154 deadline: 1680670651542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:31,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49112 deadline: 1680670651548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,638 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:31,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:31,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:31,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:31,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:31,640 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:31,640 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:31,640 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:31,640 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:31,641 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,641 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:31,642 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,642 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:31,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:31,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:31,642 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,642 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:31,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:31,642 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:31,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:31,642 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:31,643 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:31,643 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:31,643 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:31,643 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:31,643 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:31,643 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:31,643 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:31,643 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:31,643 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:31,643 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:31,643 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:31,643 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:31,643 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:31,643 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,643 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:31,643 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:31,644 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:31,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:31,644 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:31,645 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,645 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:31,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:31,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:31,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:31,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:31,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14d669ef[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:31,646 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:31,646 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,646 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,647 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:31,647 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:31,647 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:31,647 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,647 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:31,649 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:31,649 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,649 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:31,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:31,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:31,649 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:31,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:31,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:31,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:31,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:31,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:31,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:31,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:31,651 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:31,651 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:31,651 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:31,651 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:31,651 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:31,653 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,653 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:31,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:31,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:31,654 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:31,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:31,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:31,655 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:31,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:31,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:31,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,656 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:31,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:31,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,658 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:31,658 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:31,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:31,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:31,658 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:31,658 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:31,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:31,658 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:31,658 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:31,659 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:31,659 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,659 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,659 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:31,659 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:31,659 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:31,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:31,659 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,744 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:31,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49118 deadline: 1680670651744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:31,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49082 deadline: 1680670651745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:31,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49154 deadline: 1680670651746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:31,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49112 deadline: 1680670651750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,759 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:31,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:31,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:31,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:31,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:31,762 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:31,762 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:31,762 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:31,762 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:31,762 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,762 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:31,763 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:31,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:31,763 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:31,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:31,763 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,764 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:31,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:31,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:31,764 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:31,765 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:31,765 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:31,765 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:31,765 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:31,765 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:31,765 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:31,765 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:31,765 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:31,765 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:31,765 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:31,765 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:31,766 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:31,766 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:31,766 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,766 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:31,766 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:31,767 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:31,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:31,767 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,767 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:31,767 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:31,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:31,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:31,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:31,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:31,768 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:31,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56517ef8[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:31,768 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,769 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,769 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:31,769 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:31,769 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,769 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:31,769 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:31,770 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:31,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:31,770 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:31,770 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:31,770 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:31,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:31,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:31,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:31,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:31,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:31,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:31,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:31,772 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:31,772 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:31,772 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:31,772 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:31,772 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:31,773 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,773 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,773 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:31,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:31,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:31,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,774 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:31,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:31,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:31,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:31,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:31,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:31,775 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:31,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:31,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,776 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,776 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:31,776 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:31,776 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,776 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,776 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,776 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,776 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:31,777 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:31,777 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:31,777 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:31,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:31,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:31,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:31,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:31,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:31,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:31,778 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:31,778 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:31,829 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=458 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/cf0741ae0a6545ff8201b84fde965758 2023-04-05 04:56:31,846 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=458 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/09da5ad91d3d4f729174ec9fa8f832f7 2023-04-05 04:56:31,876 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=458 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/62b177f679d2419a83b0ccfca885a951 2023-04-05 04:56:31,879 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:31,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:31,882 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/cf0741ae0a6545ff8201b84fde965758 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/cf0741ae0a6545ff8201b84fde965758 2023-04-05 04:56:31,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:31,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:31,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:31,883 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:31,883 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:31,883 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:31,883 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:31,884 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,884 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:31,885 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:31,885 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:31,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:31,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:31,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:31,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:31,886 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:31,886 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:31,886 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:31,886 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:31,886 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:31,886 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:31,887 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:31,887 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:31,887 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:31,887 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:31,887 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:31,887 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:31,887 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:31,887 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:31,887 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,887 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:31,887 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:31,888 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,888 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:31,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:31,888 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,888 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:31,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:31,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:31,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:31,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:31,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:31,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c871efd[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:31,890 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,890 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:31,890 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:31,891 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,892 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,892 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:31,892 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:31,892 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:31,893 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/cf0741ae0a6545ff8201b84fde965758, entries=350, sequenceid=458, filesize=21.4 K 2023-04-05 04:56:31,895 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:31,895 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,895 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:31,895 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:31,896 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/09da5ad91d3d4f729174ec9fa8f832f7 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/09da5ad91d3d4f729174ec9fa8f832f7 2023-04-05 04:56:31,896 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:31,896 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:31,896 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:31,896 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:31,896 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:31,896 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:31,896 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:31,898 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,898 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:31,898 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:31,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:31,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:31,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:31,899 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:31,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:31,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:31,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:31,900 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:31,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:31,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,902 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,902 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:31,902 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:31,902 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:31,902 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:31,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:31,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:31,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:31,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:31,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:31,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:31,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:31,902 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:31,904 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:31,904 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,904 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:31,904 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:31,903 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:31,904 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:31,906 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/09da5ad91d3d4f729174ec9fa8f832f7, entries=150, sequenceid=458, filesize=11.9 K 2023-04-05 04:56:31,907 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/62b177f679d2419a83b0ccfca885a951 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/62b177f679d2419a83b0ccfca885a951 2023-04-05 04:56:31,912 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/62b177f679d2419a83b0ccfca885a951, entries=150, sequenceid=458, filesize=11.9 K 2023-04-05 04:56:31,915 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for ed7bbafaddb97066dca4b03f35c9eee7 in 531ms, sequenceid=458, compaction requested=true 2023-04-05 04:56:31,915 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:31,915 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:56:31,915 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:56:31,915 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:56:31,915 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:56:31,915 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:56:31,917 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 66782 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:56:31,917 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): ed7bbafaddb97066dca4b03f35c9eee7/A is initiating minor compaction (all files) 2023-04-05 04:56:31,917 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ed7bbafaddb97066dca4b03f35c9eee7/A in TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:31,917 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/739d412b65b5411bb6674a05d93e508e, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/1e123758893f4a8cb7d199af07f03db3, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/f5012106f8e54e3ea9b66c161428bf17, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/cf0741ae0a6545ff8201b84fde965758] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp, totalSize=65.2 K 2023-04-05 04:56:31,917 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49698 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:56:31,917 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): ed7bbafaddb97066dca4b03f35c9eee7/B is initiating minor compaction (all files) 2023-04-05 04:56:31,917 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ed7bbafaddb97066dca4b03f35c9eee7/B in TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:31,918 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/4ff3f16da04144d3926ca18fb79dea4f, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/9abbd6a67c1f482e989615e0a8c0ea74, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/08163ef5a2364ba1ae4c3c50fa871d6f, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/09da5ad91d3d4f729174ec9fa8f832f7] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp, totalSize=48.5 K 2023-04-05 04:56:31,918 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 739d412b65b5411bb6674a05d93e508e, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1680670590050 2023-04-05 04:56:31,918 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 4ff3f16da04144d3926ca18fb79dea4f, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1680670590050 2023-04-05 04:56:31,919 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 9abbd6a67c1f482e989615e0a8c0ea74, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1680670590132 2023-04-05 04:56:31,919 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 1e123758893f4a8cb7d199af07f03db3, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1680670590124 2023-04-05 04:56:31,919 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 08163ef5a2364ba1ae4c3c50fa871d6f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=437, earliestPutTs=1680670590228 2023-04-05 04:56:31,919 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting f5012106f8e54e3ea9b66c161428bf17, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=437, earliestPutTs=1680670590228 2023-04-05 04:56:31,920 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 09da5ad91d3d4f729174ec9fa8f832f7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=458, earliestPutTs=1680670591358 2023-04-05 04:56:31,920 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting cf0741ae0a6545ff8201b84fde965758, keycount=350, bloomtype=ROW, size=21.4 K, encoding=NONE, compression=NONE, seqNum=458, earliestPutTs=1680670591328 2023-04-05 04:56:31,936 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): ed7bbafaddb97066dca4b03f35c9eee7#A#compaction#96 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:56:31,936 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): ed7bbafaddb97066dca4b03f35c9eee7#B#compaction#95 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:56:31,977 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/a50ddd35d4b94a2bbcb4d2070e790638 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/a50ddd35d4b94a2bbcb4d2070e790638 2023-04-05 04:56:31,990 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ed7bbafaddb97066dca4b03f35c9eee7/A of ed7bbafaddb97066dca4b03f35c9eee7 into a50ddd35d4b94a2bbcb4d2070e790638(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:56:31,991 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:31,991 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7., storeName=ed7bbafaddb97066dca4b03f35c9eee7/A, priority=12, startTime=1680670591915; duration=0sec 2023-04-05 04:56:31,991 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:56:31,992 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:56:31,993 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49664 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:56:31,993 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): ed7bbafaddb97066dca4b03f35c9eee7/C is initiating minor compaction (all files) 2023-04-05 04:56:31,994 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ed7bbafaddb97066dca4b03f35c9eee7/C in TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:31,994 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/80428c7c04fc47008f7b50893a413166, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/c15659ac25af48029454ff74506a2106, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/880d44ffbc08487680a579a1f8d91419, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/62b177f679d2419a83b0ccfca885a951] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp, totalSize=48.5 K 2023-04-05 04:56:31,994 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 80428c7c04fc47008f7b50893a413166, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1680670590050 2023-04-05 04:56:31,995 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting c15659ac25af48029454ff74506a2106, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1680670590132 2023-04-05 04:56:31,995 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 880d44ffbc08487680a579a1f8d91419, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=437, earliestPutTs=1680670590228 2023-04-05 04:56:31,998 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 62b177f679d2419a83b0ccfca885a951, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=458, earliestPutTs=1680670591358 2023-04-05 04:56:32,004 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:32,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:32,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:32,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:32,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:32,008 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:32,008 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:32,008 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:32,008 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:32,009 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,009 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:32,009 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:32,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:32,009 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:32,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:32,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,011 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,011 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:32,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:32,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:32,011 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:32,021 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:32,021 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:32,021 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:32,022 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:32,022 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:32,022 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:32,022 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:32,022 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:32,022 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:32,022 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ed7bbafaddb97066dca4b03f35c9eee7 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 04:56:32,022 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=A 2023-04-05 04:56:32,023 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:32,023 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=B 2023-04-05 04:56:32,023 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:32,023 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=C 2023-04-05 04:56:32,023 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:32,027 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): ed7bbafaddb97066dca4b03f35c9eee7#C#compaction#97 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:56:32,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:32,047 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/038dd08d6e934161a58dd71b8be6328e 2023-04-05 04:56:32,046 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:32,059 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/83c91b89e6b14e7bb60877e471703b6b as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/83c91b89e6b14e7bb60877e471703b6b 2023-04-05 04:56:32,076 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ed7bbafaddb97066dca4b03f35c9eee7/C of ed7bbafaddb97066dca4b03f35c9eee7 into 83c91b89e6b14e7bb60877e471703b6b(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:56:32,076 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:32,077 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7., storeName=ed7bbafaddb97066dca4b03f35c9eee7/C, priority=12, startTime=1680670591915; duration=0sec 2023-04-05 04:56:32,077 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:56:32,099 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:32,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49154 deadline: 1680670652087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,099 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/86a71482c29843b5bf03c0e7c65399ed 2023-04-05 04:56:32,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680670592077, max timestamp=1680670592097Num uniques 859; 2023-04-05 04:56:32,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 04:56:32,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680670592077, max timestamp=1680670592097Num uniques 859; 2023-04-05 04:56:32,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 04:56:32,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680670592077, max timestamp=1680670592097Num uniques 859; 2023-04-05 04:56:32,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-05 04:56:32,140 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/b64566414a964b4b96d82612108d1787 2023-04-05 04:56:32,153 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/038dd08d6e934161a58dd71b8be6328e as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/038dd08d6e934161a58dd71b8be6328e 2023-04-05 04:56:32,155 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:32,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49112 deadline: 1680670652146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:32,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49118 deadline: 1680670652149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:32,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49082 deadline: 1680670652149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,163 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/038dd08d6e934161a58dd71b8be6328e, entries=150, sequenceid=470, filesize=11.9 K 2023-04-05 04:56:32,165 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/86a71482c29843b5bf03c0e7c65399ed as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/86a71482c29843b5bf03c0e7c65399ed 2023-04-05 04:56:32,178 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/86a71482c29843b5bf03c0e7c65399ed, entries=150, sequenceid=470, filesize=11.9 K 2023-04-05 04:56:32,179 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/b64566414a964b4b96d82612108d1787 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/b64566414a964b4b96d82612108d1787 2023-04-05 04:56:32,190 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/b64566414a964b4b96d82612108d1787, entries=150, sequenceid=470, filesize=11.9 K 2023-04-05 04:56:32,192 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=129.31 KB/132415 for ed7bbafaddb97066dca4b03f35c9eee7 in 170ms, sequenceid=470, compaction requested=false 2023-04-05 04:56:32,192 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:32,193 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:32,193 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:32,193 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:32,193 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,193 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:32,193 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:32,194 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,194 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:32,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:32,194 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,194 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:32,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:32,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:32,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:32,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:32,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:32,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63876570[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:32,201 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:32,201 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,206 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,206 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:32,206 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:32,206 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:32,206 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,206 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:32,207 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:32,207 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:32,207 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:32,207 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:32,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:32,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:32,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:32,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:32,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:32,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:32,210 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ed7bbafaddb97066dca4b03f35c9eee7 3/3 column families, dataSize=136.02 KB heapSize=384.46 KB 2023-04-05 04:56:32,212 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=A 2023-04-05 04:56:32,212 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:56:32,212 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=B 2023-04-05 04:56:32,212 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:56:32,212 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=C 2023-04-05 04:56:32,212 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:56:32,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:32,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:32,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:32,214 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:32,214 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:32,214 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:32,214 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:32,214 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:32,220 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,220 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,221 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:32,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:32,221 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:32,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:32,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:32,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:32,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:32,222 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:32,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:32,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:32,226 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,226 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:32,227 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:32,227 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:32,227 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:32,227 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:32,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:32,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:32,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:32,227 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,227 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:32,227 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:32,227 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:32,227 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:32,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:32,227 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:32,235 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:32,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49154 deadline: 1680670652235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:32,259 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:32,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49112 deadline: 1680670652258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49118 deadline: 1680670652258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,259 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:32,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49082 deadline: 1680670652258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,327 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:32,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:32,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:32,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:32,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:32,330 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:32,330 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:32,330 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:32,330 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:32,331 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,331 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:32,332 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,332 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:32,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:32,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:32,332 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,332 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:32,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:32,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:32,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:32,333 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:32,333 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:32,333 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:32,333 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:32,333 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:32,333 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:32,333 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:32,333 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:32,333 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:32,333 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:32,333 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:32,333 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:32,333 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:32,334 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:32,334 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,334 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:32,334 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:32,336 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,336 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:32,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:32,336 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,336 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:32,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:32,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:32,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:32,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:32,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:32,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4439f1fe[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:32,338 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:32,338 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,339 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,339 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:32,339 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:32,339 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:32,339 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,339 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:32,340 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:32,340 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:32,340 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:32,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:32,340 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:32,340 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:32,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:32,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49154 deadline: 1680670652336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:32,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:32,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:32,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:32,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:32,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:32,343 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:32,343 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:32,343 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:32,343 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:32,343 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:32,345 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,346 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,346 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,346 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,346 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:32,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:32,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:32,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:32,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:32,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,350 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,350 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:32,350 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:32,350 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,350 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,350 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:32,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:32,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:32,350 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:32,350 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:32,350 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:32,351 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:32,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:32,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:32,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:32,350 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,355 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:32,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:32,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:32,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:32,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:32,381 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/e4c07a91c7b546e1a6dab3bfd09fa66d as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/e4c07a91c7b546e1a6dab3bfd09fa66d 2023-04-05 04:56:32,387 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ed7bbafaddb97066dca4b03f35c9eee7/B of ed7bbafaddb97066dca4b03f35c9eee7 into e4c07a91c7b546e1a6dab3bfd09fa66d(size=13.0 K), total size for store is 24.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:56:32,387 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:32,387 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7., storeName=ed7bbafaddb97066dca4b03f35c9eee7/B, priority=12, startTime=1680670591915; duration=0sec 2023-04-05 04:56:32,387 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:56:32,451 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:32,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:32,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:32,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:32,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:32,455 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:32,455 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:32,455 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:32,455 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:32,456 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,456 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:32,456 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:32,456 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:32,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:32,457 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,457 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:32,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:32,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:32,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:32,458 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:32,462 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:32,462 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:32,462 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:32,462 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:32,462 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:32,463 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:32,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49082 deadline: 1680670652461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,464 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:32,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49118 deadline: 1680670652461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,465 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:32,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49112 deadline: 1680670652461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,474 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:32,474 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:32,474 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:32,474 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:32,474 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:32,474 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:32,474 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:32,474 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:32,474 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,475 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:32,475 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:32,476 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:32,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:32,476 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:32,476 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,476 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:32,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:32,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:32,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:32,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:32,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1942f13f[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:32,479 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:32,479 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,487 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,487 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,487 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:32,487 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:32,487 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:32,487 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:32,488 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:32,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:32,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:32,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:32,489 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:32,489 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:32,489 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:32,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:32,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:32,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:32,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:32,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:32,490 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:32,490 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:32,490 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:32,490 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:32,490 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:32,492 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,492 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:32,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:32,492 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:32,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,492 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:32,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:32,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:32,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:32,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:32,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:32,493 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:32,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:32,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,495 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,495 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:32,495 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:32,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:32,495 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:32,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:32,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:32,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:32,495 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:32,495 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:32,496 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,496 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:32,496 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,496 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:32,496 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:32,496 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:32,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:32,544 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:32,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49154 deadline: 1680670652543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,596 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:32,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:32,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:32,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:32,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:32,599 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:32,599 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:32,599 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:32,599 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:32,599 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,599 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:32,600 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:32,600 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:32,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:32,600 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,600 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:32,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:32,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:32,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:32,601 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:32,601 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:32,601 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:32,601 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:32,601 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:32,601 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:32,601 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:32,601 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:32,601 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:32,601 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:32,601 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:32,602 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:32,602 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:32,602 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:32,602 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,602 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:32,602 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:32,603 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,603 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:32,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:32,603 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,603 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:32,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:32,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:32,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:32,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:32,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:32,604 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:32,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d46741c[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:32,604 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,605 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,605 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:32,605 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:32,605 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:32,605 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,605 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:32,606 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,606 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:32,606 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:32,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:32,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:32,606 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:32,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:32,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:32,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:32,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:32,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:32,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:32,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:32,608 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:32,608 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:32,608 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:32,608 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:32,608 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:32,609 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,609 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:32,609 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:32,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:32,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:32,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:32,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:32,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,611 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:32,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:32,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:32,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,612 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:32,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:32,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,627 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=507 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/c939b92410324819b25fe00145906e0d 2023-04-05 04:56:32,629 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:32,629 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:32,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:32,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:32,629 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:32,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:32,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:32,629 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:32,629 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:32,630 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:32,629 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:32,630 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:32,630 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,630 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,630 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,630 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:32,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:32,730 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:32,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:32,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:32,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:32,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:32,732 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:32,732 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:32,732 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:32,732 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:32,733 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,733 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:32,734 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,734 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:32,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:32,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:32,734 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,734 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:32,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:32,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:32,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:32,734 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:32,735 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:32,735 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:32,735 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:32,735 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:32,735 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:32,735 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:32,735 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:32,735 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:32,735 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:32,735 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:32,735 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:32,735 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:32,735 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:32,735 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,735 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:32,736 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:32,736 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,736 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:32,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:32,737 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,737 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:32,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:32,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:32,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:32,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:32,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:32,738 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:32,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35323021[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:32,738 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,738 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,738 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:32,738 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:32,738 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:32,738 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,738 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:32,740 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:32,740 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:32,740 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:32,742 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:32,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:32,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:32,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:32,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:32,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:32,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:32,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:32,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:32,744 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:32,744 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:32,744 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:32,744 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:32,744 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:32,751 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,751 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,751 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:32,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:32,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:32,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:32,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:32,752 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:32,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:32,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:32,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:32,753 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:32,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:32,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,754 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,754 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:32,754 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:32,754 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:32,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:32,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:32,757 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:32,757 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,757 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:32,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:32,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:32,757 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,757 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,757 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:32,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:32,757 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:32,757 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:32,758 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:32,765 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:32,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49082 deadline: 1680670652764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,766 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:32,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49118 deadline: 1680670652765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,766 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:32,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49112 deadline: 1680670652766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,846 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:32,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49154 deadline: 1680670652846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,857 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:32,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:32,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:32,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:32,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:32,861 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:32,861 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:32,861 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:32,861 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:32,862 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,862 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:32,862 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:32,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:32,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:32,863 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:32,863 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,863 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:32,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:32,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:32,864 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:32,864 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:32,864 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:32,864 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:32,864 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:32,864 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:32,864 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:32,864 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:32,864 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:32,864 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:32,864 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:32,864 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:32,864 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:32,864 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:32,864 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,864 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:32,864 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:32,865 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,865 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:32,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:32,866 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:32,866 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:32,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:32,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:32,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:32,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:32,867 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:32,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e8f9178[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:32,867 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,868 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,868 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:32,868 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:32,868 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:32,868 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,868 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:32,869 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,869 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:32,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:32,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:32,869 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:32,869 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:32,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:32,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:32,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:32,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:32,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:32,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:32,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:32,871 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:32,871 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:32,871 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:32,871 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:32,871 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:32,872 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,872 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:32,872 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:32,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:32,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:32,872 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:32,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:32,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:32,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:32,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:32,873 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:32,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:32,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,875 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,875 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:32,875 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:32,875 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:32,875 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,875 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:32,875 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:32,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:32,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:32,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:32,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:32,875 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:32,875 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,876 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,876 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:32,876 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:32,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:32,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:32,976 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:32,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:32,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:32,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:32,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:32,984 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:32,984 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:32,984 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:32,984 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:32,985 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,985 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:32,985 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:32,985 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:32,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:32,986 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,986 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:32,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:32,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:32,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:32,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:32,987 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:32,987 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:32,987 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:32,987 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:32,987 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:32,987 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:32,987 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:32,987 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:32,987 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:32,987 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:32,988 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:32,988 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:32,988 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:32,988 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:32,988 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,988 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:32,988 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:32,989 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,989 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:32,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:32,989 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:32,989 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:32,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:32,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:32,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:32,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:32,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:32,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:32,994 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:32,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@304a449f[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:32,994 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:33,001 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:33,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:33,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:33,002 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:33,002 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:33,002 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,002 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:33,002 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:33,004 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:33,004 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:33,004 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:33,004 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:33,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:33,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:33,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:33,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:33,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:33,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:33,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:33,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:33,010 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:33,010 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:33,010 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:33,010 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:33,010 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:33,018 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:33,018 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:33,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:33,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:33,018 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:33,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:33,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:33,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:33,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:33,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:33,019 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:33,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:33,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:33,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:33,020 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:33,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:33,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:33,021 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:33,021 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:33,021 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:33,021 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:33,021 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:33,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:33,038 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,039 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:33,039 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:33,039 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:33,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:33,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:33,043 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:33,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:33,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:33,044 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:33,044 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:33,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:33,044 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=507 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/3323c61c1dba40b0869510c16c893530 2023-04-05 04:56:33,144 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:33,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:33,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:33,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:33,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:33,147 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:33,147 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:33,147 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:33,147 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:33,147 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:33,147 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:33,148 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:33,148 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:33,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:33,148 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,148 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:33,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:33,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:33,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:33,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:33,149 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:33,149 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:33,149 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:33,149 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:33,149 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:33,149 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:33,149 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:33,149 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:33,149 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:33,149 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:33,149 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:33,149 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:33,149 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:33,149 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,150 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:33,150 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:33,150 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,150 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:33,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:33,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:33,151 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:33,151 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:33,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:33,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:33,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:33,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:33,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:33,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ef2201e[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:33,153 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:33,153 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:33,156 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:33,156 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,156 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:33,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:33,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:33,156 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:33,156 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:33,156 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:33,157 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:33,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:33,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:33,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:33,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:33,158 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:33,158 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:33,158 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:33,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:33,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:33,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:33,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:33,159 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:33,159 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:33,159 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:33,159 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:33,159 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:33,160 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:33,160 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:33,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:33,160 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:33,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:33,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:33,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:33,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:33,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:33,160 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:33,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:33,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:33,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:33,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:33,161 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:33,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:33,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:33,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,162 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,163 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:33,163 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:33,163 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,163 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:33,163 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:33,163 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:33,163 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:33,163 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:33,163 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:33,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:33,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:33,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:33,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:33,163 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:33,163 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:33,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:33,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:33,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:33,263 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:33,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:33,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:33,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:33,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:33,267 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:33,267 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:33,267 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:33,267 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:33,267 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:33,268 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:33,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:33,269 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:33,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49118 deadline: 1680670653268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,269 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:33,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:33,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:33,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49082 deadline: 1680670653268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,269 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,269 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:33,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:33,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:33,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:33,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:33,270 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:33,272 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:33,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49112 deadline: 1680670653270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,274 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:33,274 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:33,274 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:33,274 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:33,274 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:33,280 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:33,280 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:33,281 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:33,281 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:33,282 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:33,282 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:33,282 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:33,282 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:33,282 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,282 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:33,282 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:33,286 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,286 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:33,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:33,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:33,287 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:33,287 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:33,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:33,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:33,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:33,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:33,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:33,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a65b6d9[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:33,288 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:33,288 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:33,289 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:33,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:33,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:33,289 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:33,289 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:33,289 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:33,289 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,289 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:33,289 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:33,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:33,290 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:33,290 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:33,290 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:33,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:33,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:33,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:33,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:33,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:33,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:33,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:33,292 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:33,292 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:33,292 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:33,292 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:33,292 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:33,293 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:33,293 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:33,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:33,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:33,293 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:33,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:33,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:33,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:33,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:33,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:33,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:33,294 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:33,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:33,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:33,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:33,299 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:33,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:33,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:33,306 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:33,306 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:33,306 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,306 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:33,306 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:33,306 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,306 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:33,306 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:33,306 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:33,306 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:33,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:33,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:33,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:33,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:33,307 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:33,307 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:33,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:33,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:33,349 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:33,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49154 deadline: 1680670653348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,407 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:33,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:33,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:33,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:33,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:33,413 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:33,413 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:33,414 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:33,414 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:33,414 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:33,414 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:33,416 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:33,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:33,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:33,416 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:33,416 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,416 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:33,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:33,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:33,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:33,417 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:33,418 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:33,418 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:33,418 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:33,418 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:33,418 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:33,422 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:33,422 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:33,426 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:33,426 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:33,426 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. as already flushing 2023-04-05 04:56:33,426 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:33,426 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:33,426 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:33,426 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,426 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:33,426 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:33,429 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,429 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:33,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:33,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:33,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:33,434 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:33,434 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:33,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:33,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:33,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:33,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:33,437 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:33,437 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:33,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8a3fabc[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:33,438 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:33,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:33,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:33,438 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:33,438 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:33,438 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:33,438 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,439 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:33,440 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,440 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:33,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:33,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:33,440 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:33,440 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:33,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:33,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:33,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:33,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:33,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:33,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:33,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:33,442 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:33,443 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:33,443 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:33,443 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:33,443 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:33,444 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:33,444 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:33,444 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:33,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:33,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:33,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:33,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:33,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:33,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:33,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:33,445 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:33,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:33,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:33,447 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:33,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:33,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:33,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:33,450 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:33,450 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:33,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:33,450 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:33,450 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:33,450 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:33,450 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:33,450 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,450 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:33,450 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:33,450 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:33,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:33,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:33,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:33,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:33,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:33,450 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:33,451 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:33,462 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=507 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/8a3f1ddd0bcd469a8630e1e49178eda6 2023-04-05 04:56:33,471 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/c939b92410324819b25fe00145906e0d as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/c939b92410324819b25fe00145906e0d 2023-04-05 04:56:33,482 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/c939b92410324819b25fe00145906e0d, entries=200, sequenceid=507, filesize=14.3 K 2023-04-05 04:56:33,483 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/3323c61c1dba40b0869510c16c893530 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/3323c61c1dba40b0869510c16c893530 2023-04-05 04:56:33,489 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/3323c61c1dba40b0869510c16c893530, entries=150, sequenceid=507, filesize=11.9 K 2023-04-05 04:56:33,490 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/8a3f1ddd0bcd469a8630e1e49178eda6 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/8a3f1ddd0bcd469a8630e1e49178eda6 2023-04-05 04:56:33,496 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/8a3f1ddd0bcd469a8630e1e49178eda6, entries=150, sequenceid=507, filesize=11.9 K 2023-04-05 04:56:33,497 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~136.02 KB/139285, heapSize ~384.41 KB/393640, currentSize=46.96 KB/48090 for ed7bbafaddb97066dca4b03f35c9eee7 in 1287ms, sequenceid=507, compaction requested=true 2023-04-05 04:56:33,497 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:33,497 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:56:33,497 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:56:33,497 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:56:33,497 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:56:33,497 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:56:33,498 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:56:33,498 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): ed7bbafaddb97066dca4b03f35c9eee7/B is initiating minor compaction (all files) 2023-04-05 04:56:33,498 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ed7bbafaddb97066dca4b03f35c9eee7/B in TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:33,498 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/e4c07a91c7b546e1a6dab3bfd09fa66d, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/86a71482c29843b5bf03c0e7c65399ed, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/3323c61c1dba40b0869510c16c893530] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp, totalSize=36.8 K 2023-04-05 04:56:33,499 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40088 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:56:33,499 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): ed7bbafaddb97066dca4b03f35c9eee7/A is initiating minor compaction (all files) 2023-04-05 04:56:33,499 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting e4c07a91c7b546e1a6dab3bfd09fa66d, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=458, earliestPutTs=1680670591358 2023-04-05 04:56:33,499 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ed7bbafaddb97066dca4b03f35c9eee7/A in TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:33,499 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/a50ddd35d4b94a2bbcb4d2070e790638, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/038dd08d6e934161a58dd71b8be6328e, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/c939b92410324819b25fe00145906e0d] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp, totalSize=39.1 K 2023-04-05 04:56:33,499 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 86a71482c29843b5bf03c0e7c65399ed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1680670591393 2023-04-05 04:56:33,499 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting a50ddd35d4b94a2bbcb4d2070e790638, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=458, earliestPutTs=1680670591358 2023-04-05 04:56:33,499 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 3323c61c1dba40b0869510c16c893530, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=507, earliestPutTs=1680670592124 2023-04-05 04:56:33,499 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 038dd08d6e934161a58dd71b8be6328e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1680670591393 2023-04-05 04:56:33,500 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting c939b92410324819b25fe00145906e0d, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=507, earliestPutTs=1680670592124 2023-04-05 04:56:33,511 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): ed7bbafaddb97066dca4b03f35c9eee7#B#compaction#105 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:56:33,511 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): ed7bbafaddb97066dca4b03f35c9eee7#A#compaction#104 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:56:33,551 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:33,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:33,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:33,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:33,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:33,554 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:33,554 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:33,554 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:33,554 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:33,556 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:33,557 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:33,557 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:33,557 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:33,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:33,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:33,558 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:33,558 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:33,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:33,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:33,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:33,558 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:33,559 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:33,559 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:33,559 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:33,559 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:33,559 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:33,562 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:33,562 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:33,562 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:33,562 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:33,562 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ed7bbafaddb97066dca4b03f35c9eee7 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 04:56:33,562 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=A 2023-04-05 04:56:33,563 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:33,563 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=B 2023-04-05 04:56:33,563 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:33,563 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=C 2023-04-05 04:56:33,563 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:33,590 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/d1371ddd08ca40b4ba311c6355fb839c 2023-04-05 04:56:33,618 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/f37d821f435c41c3a78b67659414c58f 2023-04-05 04:56:33,925 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/915bf2b53e2944e79c0a1085b5d0e480 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/915bf2b53e2944e79c0a1085b5d0e480 2023-04-05 04:56:33,933 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/505236a0c54a4269aed1fb2479b421b5 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/505236a0c54a4269aed1fb2479b421b5 2023-04-05 04:56:33,933 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ed7bbafaddb97066dca4b03f35c9eee7/B of ed7bbafaddb97066dca4b03f35c9eee7 into 915bf2b53e2944e79c0a1085b5d0e480(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:56:33,933 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:33,933 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7., storeName=ed7bbafaddb97066dca4b03f35c9eee7/B, priority=13, startTime=1680670593497; duration=0sec 2023-04-05 04:56:33,934 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:56:33,934 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:56:33,935 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37614 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:56:33,935 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): ed7bbafaddb97066dca4b03f35c9eee7/C is initiating minor compaction (all files) 2023-04-05 04:56:33,935 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ed7bbafaddb97066dca4b03f35c9eee7/C in TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:33,936 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/83c91b89e6b14e7bb60877e471703b6b, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/b64566414a964b4b96d82612108d1787, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/8a3f1ddd0bcd469a8630e1e49178eda6] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp, totalSize=36.7 K 2023-04-05 04:56:33,937 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 83c91b89e6b14e7bb60877e471703b6b, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=458, earliestPutTs=1680670591358 2023-04-05 04:56:33,937 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting b64566414a964b4b96d82612108d1787, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1680670591393 2023-04-05 04:56:33,938 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 8a3f1ddd0bcd469a8630e1e49178eda6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=507, earliestPutTs=1680670592124 2023-04-05 04:56:33,940 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ed7bbafaddb97066dca4b03f35c9eee7/A of ed7bbafaddb97066dca4b03f35c9eee7 into 505236a0c54a4269aed1fb2479b421b5(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:56:33,940 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:33,940 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7., storeName=ed7bbafaddb97066dca4b03f35c9eee7/A, priority=13, startTime=1680670593497; duration=0sec 2023-04-05 04:56:33,940 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:56:33,967 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): ed7bbafaddb97066dca4b03f35c9eee7#C#compaction#109 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:56:34,041 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/99c111da16cd41c8b80a2890d202ed90 2023-04-05 04:56:34,048 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/d1371ddd08ca40b4ba311c6355fb839c as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/d1371ddd08ca40b4ba311c6355fb839c 2023-04-05 04:56:34,054 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/d1371ddd08ca40b4ba311c6355fb839c, entries=150, sequenceid=518, filesize=11.9 K 2023-04-05 04:56:34,055 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/f37d821f435c41c3a78b67659414c58f as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/f37d821f435c41c3a78b67659414c58f 2023-04-05 04:56:34,060 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/f37d821f435c41c3a78b67659414c58f, entries=150, sequenceid=518, filesize=11.9 K 2023-04-05 04:56:34,061 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/99c111da16cd41c8b80a2890d202ed90 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/99c111da16cd41c8b80a2890d202ed90 2023-04-05 04:56:34,070 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/99c111da16cd41c8b80a2890d202ed90, entries=150, sequenceid=518, filesize=11.9 K 2023-04-05 04:56:34,071 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for ed7bbafaddb97066dca4b03f35c9eee7 in 509ms, sequenceid=518, compaction requested=false 2023-04-05 04:56:34,071 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:34,071 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:34,071 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:34,071 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:34,071 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:34,074 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:34,074 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:34,076 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:34,076 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:34,076 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:34,076 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:34,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:34,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:34,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:34,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:34,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:34,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:34,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:34,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:34,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:34,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a2f298b[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:34,080 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:34,080 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:34,080 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:34,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:34,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:34,081 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:34,081 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:34,081 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:34,081 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:34,081 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:34,081 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:34,081 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:34,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:34,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:34,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:34,081 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:34,082 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:34,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:34,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:34,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:34,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:34,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:34,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:34,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:34,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:34,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:34,084 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:34,084 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:34,084 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:34,084 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:34,085 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:34,086 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:34,086 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:34,086 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:34,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:34,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:34,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:34,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:34,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:34,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:34,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:34,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:34,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:34,088 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:34,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:34,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:34,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:34,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:34,089 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:34,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:34,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:34,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:34,091 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:34,091 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:34,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:34,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:34,091 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:34,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:34,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:34,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:34,091 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:34,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:34,091 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:34,091 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:34,091 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:34,091 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:34,092 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:34,092 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:34,092 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:34,092 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:34,192 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:34,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:34,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:34,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:34,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:34,195 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:34,195 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:34,195 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:34,195 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:34,195 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:34,195 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:34,196 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:34,196 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:34,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:34,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:34,196 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:34,196 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:34,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:34,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:34,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:34,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:34,197 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:34,197 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:34,197 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:34,197 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:34,197 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:34,197 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:34,197 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:34,198 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:34,198 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:34,198 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. started... 2023-04-05 04:56:34,198 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:34,198 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:34,198 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:34,198 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:34,198 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:34,198 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:34,198 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:34,199 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:34,200 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:34,200 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:34,201 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:34,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:34,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:34,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:34,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:34,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:34,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:34,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:34,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:34,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:34,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15a21600[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:34,204 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:34,204 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:34,205 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:34,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:34,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:34,205 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:34,205 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:34,205 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:34,205 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:34,205 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:34,206 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:34,206 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:34,206 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:34,206 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:34,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:34,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:34,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:34,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:34,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:34,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:34,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:34,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:34,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:34,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:34,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:34,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:34,211 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:34,211 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:34,211 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:34,211 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:34,211 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:34,220 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:34,220 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:34,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:34,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:34,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:34,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:34,220 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:34,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:34,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:34,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:34,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:34,221 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:34,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:34,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:34,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:34,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:34,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:34,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:34,222 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:34,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:34,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:34,226 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:34,226 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:34,226 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:34,226 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:34,226 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:34,226 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:34,226 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:34,226 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:34,227 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:34,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:34,227 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:34,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:34,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:34,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:34,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:34,227 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:34,227 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:34,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:34,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:34,267 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1439cba9 to 127.0.0.1:56229 2023-04-05 04:56:34,267 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6a7ef79d to 127.0.0.1:56229 2023-04-05 04:56:34,267 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:56:34,267 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:56:34,268 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3fed686f to 127.0.0.1:56229 2023-04-05 04:56:34,268 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:56:34,268 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ab9467e to 127.0.0.1:56229 2023-04-05 04:56:34,268 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:56:34,275 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4654132d to 127.0.0.1:56229 2023-04-05 04:56:34,275 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:56:34,278 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3e0ba002 to 127.0.0.1:56229 2023-04-05 04:56:34,278 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:56:34,280 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x00e18ba3 to 127.0.0.1:56229 2023-04-05 04:56:34,280 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:56:34,327 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:34,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:34,358 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x77443fb0 to 127.0.0.1:56229 2023-04-05 04:56:34,358 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:56:34,374 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x136956fd to 127.0.0.1:56229 2023-04-05 04:56:34,375 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:56:34,375 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-05 04:56:34,375 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(394): wrote 89 2023-04-05 04:56:34,375 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(394): wrote 87 2023-04-05 04:56:34,375 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(394): wrote 55 2023-04-05 04:56:34,375 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(394): wrote 105 2023-04-05 04:56:34,376 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(394): wrote 75 2023-04-05 04:56:34,376 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-05 04:56:34,376 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(398): read 4289 2023-04-05 04:56:34,376 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(398): read 4302 2023-04-05 04:56:34,376 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-05 04:56:34,376 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(402): scanned 1900 2023-04-05 04:56:34,376 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(403): verified 5700 rows 2023-04-05 04:56:34,376 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(402): scanned 1914 2023-04-05 04:56:34,376 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(403): verified 5742 rows 2023-04-05 04:56:34,377 INFO [Listener at localhost.localdomain/34579] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-05 04:56:34,377 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x12582a26 to 127.0.0.1:56229 2023-04-05 04:56:34,377 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:56:34,383 INFO [Listener at localhost.localdomain/34579] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-05 04:56:34,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-05 04:56:34,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-05 04:56:34,398 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680670594398"}]},"ts":"1680670594398"} 2023-04-05 04:56:34,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-05 04:56:34,399 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-05 04:56:34,400 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/235336fd22be4a7eb51a9f58e2fec54f as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/235336fd22be4a7eb51a9f58e2fec54f 2023-04-05 04:56:34,400 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-05 04:56:34,402 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ed7bbafaddb97066dca4b03f35c9eee7, UNASSIGN}] 2023-04-05 04:56:34,404 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ed7bbafaddb97066dca4b03f35c9eee7, UNASSIGN 2023-04-05 04:56:34,405 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=ed7bbafaddb97066dca4b03f35c9eee7, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:34,405 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670594405"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680670594405"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680670594405"}]},"ts":"1680670594405"} 2023-04-05 04:56:34,406 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ed7bbafaddb97066dca4b03f35c9eee7/C of ed7bbafaddb97066dca4b03f35c9eee7 into 235336fd22be4a7eb51a9f58e2fec54f(size=13.0 K), total size for store is 24.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:56:34,406 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:34,406 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7., storeName=ed7bbafaddb97066dca4b03f35c9eee7/C, priority=13, startTime=1680670593497; duration=0sec 2023-04-05 04:56:34,406 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:56:34,406 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206}] 2023-04-05 04:56:34,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-05 04:56:34,567 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:34,567 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing ed7bbafaddb97066dca4b03f35c9eee7, disabling compactions & flushes 2023-04-05 04:56:34,567 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:34,567 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:34,567 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. after waiting 0 ms 2023-04-05 04:56:34,567 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:34,567 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing ed7bbafaddb97066dca4b03f35c9eee7 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-05 04:56:34,568 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=A 2023-04-05 04:56:34,568 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:34,568 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=B 2023-04-05 04:56:34,568 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:34,568 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed7bbafaddb97066dca4b03f35c9eee7, store=C 2023-04-05 04:56:34,568 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:34,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-05 04:56:34,978 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=530 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/bff587126526440a8785128304377ed1 2023-04-05 04:56:35,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-05 04:56:35,394 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=530 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/f6a9a89cc0ff44dd8927d98b8608716c 2023-04-05 04:56:35,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-05 04:56:35,806 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=530 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/8de27127e076477d8a8e22011b8cf29e 2023-04-05 04:56:35,818 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/A/bff587126526440a8785128304377ed1 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/bff587126526440a8785128304377ed1 2023-04-05 04:56:35,826 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/bff587126526440a8785128304377ed1, entries=150, sequenceid=530, filesize=11.9 K 2023-04-05 04:56:35,827 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/B/f6a9a89cc0ff44dd8927d98b8608716c as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/f6a9a89cc0ff44dd8927d98b8608716c 2023-04-05 04:56:35,832 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/f6a9a89cc0ff44dd8927d98b8608716c, entries=150, sequenceid=530, filesize=11.9 K 2023-04-05 04:56:35,833 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/.tmp/C/8de27127e076477d8a8e22011b8cf29e as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/8de27127e076477d8a8e22011b8cf29e 2023-04-05 04:56:35,837 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/8de27127e076477d8a8e22011b8cf29e, entries=150, sequenceid=530, filesize=11.9 K 2023-04-05 04:56:35,838 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for ed7bbafaddb97066dca4b03f35c9eee7 in 1271ms, sequenceid=530, compaction requested=true 2023-04-05 04:56:35,862 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/430c0819709b4c71aa086159941e048e, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/c13d76fee5d1496aa3ca21822f2f5a1b, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/c3b3f3f129ac4533bf397e3631b931d7, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/fb409db9b41845e38392a2ee1419572c, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/73a07b02fb1a49369267c6fdef4831fd, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/88ff7f71ed4045138801c3526c432bdd, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/e111ab7db17f4fedb8895d2162e9a8fc, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/b0225d37f88d4f708fc6f6a60a62c694, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/c6488e8bc7fe41e58638bab84861f173, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/0fca916428ce4e7dae6f208ac9f432f2, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/681e655b6f1b42c5a83d07ea1fc933d9, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/bc8a097904dc44dab24e1828eb2a9dc7, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/1bbc70f0ac1542c8b14a2ba03cd702f3, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/6cb305b077864af48242ff2c437b7121, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/fe0d8cc00b4c46e887bf34568f9018ed, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/1ef80b8838904694b9f3ededc4b4db58, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/9e60b08dc6f04d90b2225cbc01722dd9, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/9fd030343e40449bbf456aa41a03df63, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/6b08b8b81e4745278966b3ea4cf33e9a, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/13fecde8a5de48d6a9eaa4117052cc43, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/20aa8e818948457b9d56874a17efeaef, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/e464dfa7666943379ae0b788e3315c54, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/4d0ab75fc6384108b66cf9b0e23aa4cb, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/5bf09ea8b0a544359c430e5a84486fd0, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/a9dbbf169b7a439585ea201f6f2a47ce, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/1a120935c3904216bd751d558c07d002, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/4c9f904899c34d79ae20005ac5309f57, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/edd32493d15a468185f74f510d33f537, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/739d412b65b5411bb6674a05d93e508e, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/1e123758893f4a8cb7d199af07f03db3, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/f5012106f8e54e3ea9b66c161428bf17, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/cf0741ae0a6545ff8201b84fde965758, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/a50ddd35d4b94a2bbcb4d2070e790638, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/038dd08d6e934161a58dd71b8be6328e, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/c939b92410324819b25fe00145906e0d] to archive 2023-04-05 04:56:35,865 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 04:56:35,871 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/430c0819709b4c71aa086159941e048e to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/430c0819709b4c71aa086159941e048e 2023-04-05 04:56:35,873 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/c13d76fee5d1496aa3ca21822f2f5a1b to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/c13d76fee5d1496aa3ca21822f2f5a1b 2023-04-05 04:56:35,875 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/c3b3f3f129ac4533bf397e3631b931d7 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/c3b3f3f129ac4533bf397e3631b931d7 2023-04-05 04:56:35,877 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/fb409db9b41845e38392a2ee1419572c to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/fb409db9b41845e38392a2ee1419572c 2023-04-05 04:56:35,879 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/73a07b02fb1a49369267c6fdef4831fd to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/73a07b02fb1a49369267c6fdef4831fd 2023-04-05 04:56:35,881 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/88ff7f71ed4045138801c3526c432bdd to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/88ff7f71ed4045138801c3526c432bdd 2023-04-05 04:56:35,883 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/e111ab7db17f4fedb8895d2162e9a8fc to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/e111ab7db17f4fedb8895d2162e9a8fc 2023-04-05 04:56:35,884 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/b0225d37f88d4f708fc6f6a60a62c694 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/b0225d37f88d4f708fc6f6a60a62c694 2023-04-05 04:56:35,886 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/c6488e8bc7fe41e58638bab84861f173 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/c6488e8bc7fe41e58638bab84861f173 2023-04-05 04:56:35,887 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/0fca916428ce4e7dae6f208ac9f432f2 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/0fca916428ce4e7dae6f208ac9f432f2 2023-04-05 04:56:35,888 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/681e655b6f1b42c5a83d07ea1fc933d9 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/681e655b6f1b42c5a83d07ea1fc933d9 2023-04-05 04:56:35,889 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/bc8a097904dc44dab24e1828eb2a9dc7 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/bc8a097904dc44dab24e1828eb2a9dc7 2023-04-05 04:56:35,891 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/1bbc70f0ac1542c8b14a2ba03cd702f3 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/1bbc70f0ac1542c8b14a2ba03cd702f3 2023-04-05 04:56:35,892 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/6cb305b077864af48242ff2c437b7121 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/6cb305b077864af48242ff2c437b7121 2023-04-05 04:56:35,893 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/fe0d8cc00b4c46e887bf34568f9018ed to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/fe0d8cc00b4c46e887bf34568f9018ed 2023-04-05 04:56:35,895 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/1ef80b8838904694b9f3ededc4b4db58 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/1ef80b8838904694b9f3ededc4b4db58 2023-04-05 04:56:35,896 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/9e60b08dc6f04d90b2225cbc01722dd9 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/9e60b08dc6f04d90b2225cbc01722dd9 2023-04-05 04:56:35,897 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/9fd030343e40449bbf456aa41a03df63 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/9fd030343e40449bbf456aa41a03df63 2023-04-05 04:56:35,897 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/6b08b8b81e4745278966b3ea4cf33e9a to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/6b08b8b81e4745278966b3ea4cf33e9a 2023-04-05 04:56:35,899 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/13fecde8a5de48d6a9eaa4117052cc43 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/13fecde8a5de48d6a9eaa4117052cc43 2023-04-05 04:56:35,900 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/20aa8e818948457b9d56874a17efeaef to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/20aa8e818948457b9d56874a17efeaef 2023-04-05 04:56:35,901 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/e464dfa7666943379ae0b788e3315c54 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/e464dfa7666943379ae0b788e3315c54 2023-04-05 04:56:35,902 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/4d0ab75fc6384108b66cf9b0e23aa4cb to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/4d0ab75fc6384108b66cf9b0e23aa4cb 2023-04-05 04:56:35,903 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/5bf09ea8b0a544359c430e5a84486fd0 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/5bf09ea8b0a544359c430e5a84486fd0 2023-04-05 04:56:35,905 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/a9dbbf169b7a439585ea201f6f2a47ce to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/a9dbbf169b7a439585ea201f6f2a47ce 2023-04-05 04:56:35,906 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/1a120935c3904216bd751d558c07d002 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/1a120935c3904216bd751d558c07d002 2023-04-05 04:56:35,907 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/4c9f904899c34d79ae20005ac5309f57 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/4c9f904899c34d79ae20005ac5309f57 2023-04-05 04:56:35,908 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/edd32493d15a468185f74f510d33f537 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/edd32493d15a468185f74f510d33f537 2023-04-05 04:56:35,909 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/739d412b65b5411bb6674a05d93e508e to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/739d412b65b5411bb6674a05d93e508e 2023-04-05 04:56:35,910 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/1e123758893f4a8cb7d199af07f03db3 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/1e123758893f4a8cb7d199af07f03db3 2023-04-05 04:56:35,911 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/f5012106f8e54e3ea9b66c161428bf17 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/f5012106f8e54e3ea9b66c161428bf17 2023-04-05 04:56:35,912 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/cf0741ae0a6545ff8201b84fde965758 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/cf0741ae0a6545ff8201b84fde965758 2023-04-05 04:56:35,912 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/a50ddd35d4b94a2bbcb4d2070e790638 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/a50ddd35d4b94a2bbcb4d2070e790638 2023-04-05 04:56:35,913 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/038dd08d6e934161a58dd71b8be6328e to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/038dd08d6e934161a58dd71b8be6328e 2023-04-05 04:56:35,914 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/c939b92410324819b25fe00145906e0d to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/c939b92410324819b25fe00145906e0d 2023-04-05 04:56:35,936 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/046d7cbf21584d27ae1ec2fd07dfe460, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/177f9cdd9dff4ad081e3409229f4ff64, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/5d110654783444028eba632aaff4444b, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/84550ac17ed549ae9ee1d88f8695f273, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/73317ffc664844d39df8dfa8f641baf0, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/edf70cf7007b419ca0e89cc06925b694, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/9daee39e174d47e98c61a84724a3dbf3, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/74bc59a9bd5c4ca0a6dacacbf6b73c31, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/1a9555663ff8406c8c5a17b5cc0a430f, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/c8b91a51144a4c289ef51a993b16f1bc, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/78005fe6505d4c23b7b2c5cee317cbea, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/1df758f917c64c86824b85e2284178de, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/a5032e3edd9746f29de2e6f00dba272f, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/f75054eb216948b8a5812b01ae9a11f7, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/589f3fba331e47449f8f5d6781db8d94, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/717d10fece2f4089a1c4fa26e76802e2, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/220c5f4393e44d438cf105e56728f37a, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/9864b9f055044843b16a6ad6bd0179fd, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/3a97ec790d354ff4b0f1a44b15a59bc1, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/e47286b227bc4e0db8b4c4e24fe3c14d, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/25b5f2ef18234268ab6525e68f41ee9a, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/d04cc117816545fea7621d1ab431aa53, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/2ad0fde45b8547b9aa03cf6e685f965d, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/32f830c6270c4a9f884a9d573f476e7c, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/bcd4376586c94a3cb87a141ce34ad659, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/b1941cae3d7b4392ac24e48819576222, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/8202fbbd4d304d8e8246d19d967b4bd0, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/4ff3f16da04144d3926ca18fb79dea4f, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/aa80bd06b5934f94a28df180dce756d4, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/9abbd6a67c1f482e989615e0a8c0ea74, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/08163ef5a2364ba1ae4c3c50fa871d6f, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/e4c07a91c7b546e1a6dab3bfd09fa66d, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/09da5ad91d3d4f729174ec9fa8f832f7, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/86a71482c29843b5bf03c0e7c65399ed, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/3323c61c1dba40b0869510c16c893530] to archive 2023-04-05 04:56:35,937 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 04:56:35,938 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/046d7cbf21584d27ae1ec2fd07dfe460 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/046d7cbf21584d27ae1ec2fd07dfe460 2023-04-05 04:56:35,939 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/177f9cdd9dff4ad081e3409229f4ff64 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/177f9cdd9dff4ad081e3409229f4ff64 2023-04-05 04:56:35,940 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/5d110654783444028eba632aaff4444b to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/5d110654783444028eba632aaff4444b 2023-04-05 04:56:35,941 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/84550ac17ed549ae9ee1d88f8695f273 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/84550ac17ed549ae9ee1d88f8695f273 2023-04-05 04:56:35,942 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/73317ffc664844d39df8dfa8f641baf0 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/73317ffc664844d39df8dfa8f641baf0 2023-04-05 04:56:35,943 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/edf70cf7007b419ca0e89cc06925b694 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/edf70cf7007b419ca0e89cc06925b694 2023-04-05 04:56:35,945 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/9daee39e174d47e98c61a84724a3dbf3 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/9daee39e174d47e98c61a84724a3dbf3 2023-04-05 04:56:35,946 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/74bc59a9bd5c4ca0a6dacacbf6b73c31 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/74bc59a9bd5c4ca0a6dacacbf6b73c31 2023-04-05 04:56:35,947 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/1a9555663ff8406c8c5a17b5cc0a430f to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/1a9555663ff8406c8c5a17b5cc0a430f 2023-04-05 04:56:35,948 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/c8b91a51144a4c289ef51a993b16f1bc to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/c8b91a51144a4c289ef51a993b16f1bc 2023-04-05 04:56:35,949 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/78005fe6505d4c23b7b2c5cee317cbea to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/78005fe6505d4c23b7b2c5cee317cbea 2023-04-05 04:56:35,950 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/1df758f917c64c86824b85e2284178de to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/1df758f917c64c86824b85e2284178de 2023-04-05 04:56:35,951 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/a5032e3edd9746f29de2e6f00dba272f to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/a5032e3edd9746f29de2e6f00dba272f 2023-04-05 04:56:35,952 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/f75054eb216948b8a5812b01ae9a11f7 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/f75054eb216948b8a5812b01ae9a11f7 2023-04-05 04:56:35,953 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/589f3fba331e47449f8f5d6781db8d94 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/589f3fba331e47449f8f5d6781db8d94 2023-04-05 04:56:35,954 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/717d10fece2f4089a1c4fa26e76802e2 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/717d10fece2f4089a1c4fa26e76802e2 2023-04-05 04:56:35,955 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/220c5f4393e44d438cf105e56728f37a to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/220c5f4393e44d438cf105e56728f37a 2023-04-05 04:56:35,956 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/9864b9f055044843b16a6ad6bd0179fd to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/9864b9f055044843b16a6ad6bd0179fd 2023-04-05 04:56:35,957 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/3a97ec790d354ff4b0f1a44b15a59bc1 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/3a97ec790d354ff4b0f1a44b15a59bc1 2023-04-05 04:56:35,958 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/e47286b227bc4e0db8b4c4e24fe3c14d to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/e47286b227bc4e0db8b4c4e24fe3c14d 2023-04-05 04:56:35,959 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/25b5f2ef18234268ab6525e68f41ee9a to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/25b5f2ef18234268ab6525e68f41ee9a 2023-04-05 04:56:35,960 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/d04cc117816545fea7621d1ab431aa53 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/d04cc117816545fea7621d1ab431aa53 2023-04-05 04:56:35,962 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/2ad0fde45b8547b9aa03cf6e685f965d to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/2ad0fde45b8547b9aa03cf6e685f965d 2023-04-05 04:56:35,963 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/32f830c6270c4a9f884a9d573f476e7c to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/32f830c6270c4a9f884a9d573f476e7c 2023-04-05 04:56:35,964 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/bcd4376586c94a3cb87a141ce34ad659 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/bcd4376586c94a3cb87a141ce34ad659 2023-04-05 04:56:35,965 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/b1941cae3d7b4392ac24e48819576222 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/b1941cae3d7b4392ac24e48819576222 2023-04-05 04:56:35,966 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/8202fbbd4d304d8e8246d19d967b4bd0 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/8202fbbd4d304d8e8246d19d967b4bd0 2023-04-05 04:56:35,967 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/4ff3f16da04144d3926ca18fb79dea4f to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/4ff3f16da04144d3926ca18fb79dea4f 2023-04-05 04:56:35,968 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/aa80bd06b5934f94a28df180dce756d4 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/aa80bd06b5934f94a28df180dce756d4 2023-04-05 04:56:35,969 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/9abbd6a67c1f482e989615e0a8c0ea74 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/9abbd6a67c1f482e989615e0a8c0ea74 2023-04-05 04:56:35,970 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/08163ef5a2364ba1ae4c3c50fa871d6f to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/08163ef5a2364ba1ae4c3c50fa871d6f 2023-04-05 04:56:35,971 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/e4c07a91c7b546e1a6dab3bfd09fa66d to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/e4c07a91c7b546e1a6dab3bfd09fa66d 2023-04-05 04:56:35,972 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/09da5ad91d3d4f729174ec9fa8f832f7 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/09da5ad91d3d4f729174ec9fa8f832f7 2023-04-05 04:56:35,973 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/86a71482c29843b5bf03c0e7c65399ed to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/86a71482c29843b5bf03c0e7c65399ed 2023-04-05 04:56:35,974 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/3323c61c1dba40b0869510c16c893530 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/3323c61c1dba40b0869510c16c893530 2023-04-05 04:56:35,978 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/4c4cd0a96b4f46178a224a8ab1828bd8, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/f2f5a0ad425f475dace0da5f5407d77a, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/3ad1ae4863a84c638f6607b8722cbceb, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/5b7a6109cea64ff29db470e06c2dfbcc, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/e21c5139e2ce48aa8df5b79215df8bec, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/4d52be4332e543e4a5e1a22eadeb1d32, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/0deb9640a91c48fe8f232923965cfca4, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/e5d2fa9b0e9f4c6c98dd56941987eca6, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/1a4f37425f7a462f844cdd969022a343, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/7858562f20f44824a178b56731681416, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/9db0f55c83f941109c2f393aa1bbb904, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/9ae8be7063fd4972a5def3b473d7f4f2, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/381717362efc439fae4b2fa7bbce368e, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/798aff70a68e4034aad6d62b194ec086, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/2e9f0aece26c42f4a526739a3c1fe5ea, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/b80dab0cb2ae4152ab07e01fc8248a5a, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/f3a5d04f8ea7482ead6e0e85b07928ad, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/9ba8866d9ac7433c97f587502ebac0cd, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/b83ee6fe57214905a966135ddff1a7cc, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/321f0b3dd8774c42a5f092dc5d57ca14, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/b46ea9c5fb4141c6b3c3dcc8684bcfef, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/ffeb9c2d4e834763ba4be2739e108c35, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/1dfa9e9a00fa479798db87e50d830975, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/c63988d0350741c88af1a98362db12c8, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/a9f332e8e2dc454d94c99c05e0a3af83, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/ef3215a69a6942dfad75f6f4647a5a4b, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/80428c7c04fc47008f7b50893a413166, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/ce159ded61874157940369cb59467671, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/c15659ac25af48029454ff74506a2106, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/880d44ffbc08487680a579a1f8d91419, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/83c91b89e6b14e7bb60877e471703b6b, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/62b177f679d2419a83b0ccfca885a951, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/b64566414a964b4b96d82612108d1787, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/8a3f1ddd0bcd469a8630e1e49178eda6] to archive 2023-04-05 04:56:35,980 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 04:56:35,982 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/4c4cd0a96b4f46178a224a8ab1828bd8 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/4c4cd0a96b4f46178a224a8ab1828bd8 2023-04-05 04:56:35,983 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/f2f5a0ad425f475dace0da5f5407d77a to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/f2f5a0ad425f475dace0da5f5407d77a 2023-04-05 04:56:35,984 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/3ad1ae4863a84c638f6607b8722cbceb to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/3ad1ae4863a84c638f6607b8722cbceb 2023-04-05 04:56:35,986 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/5b7a6109cea64ff29db470e06c2dfbcc to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/5b7a6109cea64ff29db470e06c2dfbcc 2023-04-05 04:56:35,987 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/e21c5139e2ce48aa8df5b79215df8bec to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/e21c5139e2ce48aa8df5b79215df8bec 2023-04-05 04:56:35,988 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/4d52be4332e543e4a5e1a22eadeb1d32 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/4d52be4332e543e4a5e1a22eadeb1d32 2023-04-05 04:56:35,989 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/0deb9640a91c48fe8f232923965cfca4 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/0deb9640a91c48fe8f232923965cfca4 2023-04-05 04:56:35,991 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/e5d2fa9b0e9f4c6c98dd56941987eca6 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/e5d2fa9b0e9f4c6c98dd56941987eca6 2023-04-05 04:56:35,992 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/1a4f37425f7a462f844cdd969022a343 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/1a4f37425f7a462f844cdd969022a343 2023-04-05 04:56:35,994 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/7858562f20f44824a178b56731681416 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/7858562f20f44824a178b56731681416 2023-04-05 04:56:35,996 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/9db0f55c83f941109c2f393aa1bbb904 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/9db0f55c83f941109c2f393aa1bbb904 2023-04-05 04:56:35,998 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/9ae8be7063fd4972a5def3b473d7f4f2 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/9ae8be7063fd4972a5def3b473d7f4f2 2023-04-05 04:56:35,999 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/381717362efc439fae4b2fa7bbce368e to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/381717362efc439fae4b2fa7bbce368e 2023-04-05 04:56:36,001 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/798aff70a68e4034aad6d62b194ec086 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/798aff70a68e4034aad6d62b194ec086 2023-04-05 04:56:36,002 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/2e9f0aece26c42f4a526739a3c1fe5ea to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/2e9f0aece26c42f4a526739a3c1fe5ea 2023-04-05 04:56:36,004 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/b80dab0cb2ae4152ab07e01fc8248a5a to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/b80dab0cb2ae4152ab07e01fc8248a5a 2023-04-05 04:56:36,005 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/f3a5d04f8ea7482ead6e0e85b07928ad to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/f3a5d04f8ea7482ead6e0e85b07928ad 2023-04-05 04:56:36,007 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/9ba8866d9ac7433c97f587502ebac0cd to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/9ba8866d9ac7433c97f587502ebac0cd 2023-04-05 04:56:36,009 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/b83ee6fe57214905a966135ddff1a7cc to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/b83ee6fe57214905a966135ddff1a7cc 2023-04-05 04:56:36,010 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/321f0b3dd8774c42a5f092dc5d57ca14 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/321f0b3dd8774c42a5f092dc5d57ca14 2023-04-05 04:56:36,012 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/b46ea9c5fb4141c6b3c3dcc8684bcfef to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/b46ea9c5fb4141c6b3c3dcc8684bcfef 2023-04-05 04:56:36,014 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/ffeb9c2d4e834763ba4be2739e108c35 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/ffeb9c2d4e834763ba4be2739e108c35 2023-04-05 04:56:36,015 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/1dfa9e9a00fa479798db87e50d830975 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/1dfa9e9a00fa479798db87e50d830975 2023-04-05 04:56:36,017 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/c63988d0350741c88af1a98362db12c8 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/c63988d0350741c88af1a98362db12c8 2023-04-05 04:56:36,019 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/a9f332e8e2dc454d94c99c05e0a3af83 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/a9f332e8e2dc454d94c99c05e0a3af83 2023-04-05 04:56:36,020 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/ef3215a69a6942dfad75f6f4647a5a4b to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/ef3215a69a6942dfad75f6f4647a5a4b 2023-04-05 04:56:36,022 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/80428c7c04fc47008f7b50893a413166 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/80428c7c04fc47008f7b50893a413166 2023-04-05 04:56:36,023 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/ce159ded61874157940369cb59467671 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/ce159ded61874157940369cb59467671 2023-04-05 04:56:36,025 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/c15659ac25af48029454ff74506a2106 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/c15659ac25af48029454ff74506a2106 2023-04-05 04:56:36,026 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/880d44ffbc08487680a579a1f8d91419 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/880d44ffbc08487680a579a1f8d91419 2023-04-05 04:56:36,028 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/83c91b89e6b14e7bb60877e471703b6b to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/83c91b89e6b14e7bb60877e471703b6b 2023-04-05 04:56:36,030 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/62b177f679d2419a83b0ccfca885a951 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/62b177f679d2419a83b0ccfca885a951 2023-04-05 04:56:36,031 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/b64566414a964b4b96d82612108d1787 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/b64566414a964b4b96d82612108d1787 2023-04-05 04:56:36,033 DEBUG [StoreCloser-TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/8a3f1ddd0bcd469a8630e1e49178eda6 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/8a3f1ddd0bcd469a8630e1e49178eda6 2023-04-05 04:56:36,040 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/recovered.edits/533.seqid, newMaxSeqId=533, maxSeqId=1 2023-04-05 04:56:36,041 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7. 2023-04-05 04:56:36,042 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for ed7bbafaddb97066dca4b03f35c9eee7: 2023-04-05 04:56:36,044 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:36,045 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=ed7bbafaddb97066dca4b03f35c9eee7, regionState=CLOSED 2023-04-05 04:56:36,045 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670596045"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680670596045"}]},"ts":"1680670596045"} 2023-04-05 04:56:36,049 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-05 04:56:36,050 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure ed7bbafaddb97066dca4b03f35c9eee7, server=jenkins-hbase20.apache.org,38091,1680670569206 in 1.6410 sec 2023-04-05 04:56:36,052 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-05 04:56:36,053 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=ed7bbafaddb97066dca4b03f35c9eee7, UNASSIGN in 1.6480 sec 2023-04-05 04:56:36,054 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680670596053"}]},"ts":"1680670596053"} 2023-04-05 04:56:36,055 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-05 04:56:36,056 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-05 04:56:36,058 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.6680 sec 2023-04-05 04:56:36,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-05 04:56:36,507 INFO [Listener at localhost.localdomain/34579] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-05 04:56:36,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-05 04:56:36,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 04:56:36,525 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 04:56:36,527 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 04:56:36,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-05 04:56:36,533 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:36,536 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A, FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B, FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C, FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/recovered.edits] 2023-04-05 04:56:36,539 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/505236a0c54a4269aed1fb2479b421b5 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/505236a0c54a4269aed1fb2479b421b5 2023-04-05 04:56:36,541 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/bff587126526440a8785128304377ed1 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/bff587126526440a8785128304377ed1 2023-04-05 04:56:36,543 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/d1371ddd08ca40b4ba311c6355fb839c to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/A/d1371ddd08ca40b4ba311c6355fb839c 2023-04-05 04:56:36,545 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/915bf2b53e2944e79c0a1085b5d0e480 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/915bf2b53e2944e79c0a1085b5d0e480 2023-04-05 04:56:36,546 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/f37d821f435c41c3a78b67659414c58f to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/f37d821f435c41c3a78b67659414c58f 2023-04-05 04:56:36,548 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/f6a9a89cc0ff44dd8927d98b8608716c to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/B/f6a9a89cc0ff44dd8927d98b8608716c 2023-04-05 04:56:36,550 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/235336fd22be4a7eb51a9f58e2fec54f to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/235336fd22be4a7eb51a9f58e2fec54f 2023-04-05 04:56:36,551 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/8de27127e076477d8a8e22011b8cf29e to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/8de27127e076477d8a8e22011b8cf29e 2023-04-05 04:56:36,552 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/99c111da16cd41c8b80a2890d202ed90 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/C/99c111da16cd41c8b80a2890d202ed90 2023-04-05 04:56:36,555 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/recovered.edits/533.seqid to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7/recovered.edits/533.seqid 2023-04-05 04:56:36,555 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/ed7bbafaddb97066dca4b03f35c9eee7 2023-04-05 04:56:36,555 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-05 04:56:36,558 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 04:56:36,574 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-05 04:56:36,619 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-05 04:56:36,620 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 04:56:36,620 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-05 04:56:36,621 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680670596621"}]},"ts":"9223372036854775807"} 2023-04-05 04:56:36,623 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-05 04:56:36,623 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => ed7bbafaddb97066dca4b03f35c9eee7, NAME => 'TestAcidGuarantees,,1680670572959.ed7bbafaddb97066dca4b03f35c9eee7.', STARTKEY => '', ENDKEY => ''}] 2023-04-05 04:56:36,623 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-05 04:56:36,624 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680670596623"}]},"ts":"9223372036854775807"} 2023-04-05 04:56:36,626 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-05 04:56:36,627 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 04:56:36,629 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 109 msec 2023-04-05 04:56:36,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-05 04:56:36,633 INFO [Listener at localhost.localdomain/34579] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-05 04:56:36,653 INFO [Listener at localhost.localdomain/34579] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=237 (was 207) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-543218053_15 at /127.0.0.1:53828 [Waiting for operation #5] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1dc5d24e-shared-pool-13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase20:38091-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-543218053_15 at /127.0.0.1:53940 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1dc5d24e-shared-pool-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1dc5d24e-shared-pool-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1dc5d24e-shared-pool-11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-543218053_15 at /127.0.0.1:53930 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1dc5d24e-shared-pool-12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-543218053_15 at /127.0.0.1:53986 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=581 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=324 (was 188) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 167) - ProcessCount LEAK? -, AvailableMemoryMB=6518 (was 7049) 2023-04-05 04:56:36,664 INFO [Listener at localhost.localdomain/34579] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=237, OpenFileDescriptor=581, MaxFileDescriptor=60000, SystemLoadAverage=324, ProcessCount=168, AvailableMemoryMB=6518 2023-04-05 04:56:36,666 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-05 04:56:36,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-05 04:56:36,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-05 04:56:36,669 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-05 04:56:36,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-05 04:56:36,670 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-05 04:56:36,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-05 04:56:36,671 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:36,672 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa empty. 2023-04-05 04:56:36,672 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:36,672 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-05 04:56:36,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-05 04:56:36,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-05 04:56:37,084 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-05 04:56:37,090 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 0a496ef74db8da540124a4d88d6e37aa, NAME => 'TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp 2023-04-05 04:56:37,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-05 04:56:37,499 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 04:56:37,499 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 0a496ef74db8da540124a4d88d6e37aa, disabling compactions & flushes 2023-04-05 04:56:37,499 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:37,499 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:37,499 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. after waiting 0 ms 2023-04-05 04:56:37,499 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:37,499 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:37,499 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:56:37,502 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-05 04:56:37,504 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670597503"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680670597503"}]},"ts":"1680670597503"} 2023-04-05 04:56:37,505 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-05 04:56:37,507 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-05 04:56:37,507 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680670597507"}]},"ts":"1680670597507"} 2023-04-05 04:56:37,509 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-05 04:56:37,512 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0a496ef74db8da540124a4d88d6e37aa, ASSIGN}] 2023-04-05 04:56:37,514 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0a496ef74db8da540124a4d88d6e37aa, ASSIGN 2023-04-05 04:56:37,515 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=0a496ef74db8da540124a4d88d6e37aa, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,38091,1680670569206; forceNewPlan=false, retain=false 2023-04-05 04:56:37,667 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=0a496ef74db8da540124a4d88d6e37aa, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:37,667 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670597667"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680670597667"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680670597667"}]},"ts":"1680670597667"} 2023-04-05 04:56:37,671 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206}] 2023-04-05 04:56:37,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-05 04:56:37,828 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:37,828 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 0a496ef74db8da540124a4d88d6e37aa, NAME => 'TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.', STARTKEY => '', ENDKEY => ''} 2023-04-05 04:56:37,829 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:37,829 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 04:56:37,829 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:37,829 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:37,830 INFO [StoreOpener-0a496ef74db8da540124a4d88d6e37aa-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:37,832 DEBUG [StoreOpener-0a496ef74db8da540124a4d88d6e37aa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A 2023-04-05 04:56:37,832 DEBUG [StoreOpener-0a496ef74db8da540124a4d88d6e37aa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A 2023-04-05 04:56:37,832 INFO [StoreOpener-0a496ef74db8da540124a4d88d6e37aa-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 04:56:37,833 INFO [StoreOpener-0a496ef74db8da540124a4d88d6e37aa-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0a496ef74db8da540124a4d88d6e37aa columnFamilyName A 2023-04-05 04:56:37,833 INFO [StoreOpener-0a496ef74db8da540124a4d88d6e37aa-1] regionserver.HStore(310): Store=0a496ef74db8da540124a4d88d6e37aa/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 04:56:37,834 INFO [StoreOpener-0a496ef74db8da540124a4d88d6e37aa-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:37,836 DEBUG [StoreOpener-0a496ef74db8da540124a4d88d6e37aa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B 2023-04-05 04:56:37,836 DEBUG [StoreOpener-0a496ef74db8da540124a4d88d6e37aa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B 2023-04-05 04:56:37,836 INFO [StoreOpener-0a496ef74db8da540124a4d88d6e37aa-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 04:56:37,836 INFO [StoreOpener-0a496ef74db8da540124a4d88d6e37aa-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0a496ef74db8da540124a4d88d6e37aa columnFamilyName B 2023-04-05 04:56:37,837 INFO [StoreOpener-0a496ef74db8da540124a4d88d6e37aa-1] regionserver.HStore(310): Store=0a496ef74db8da540124a4d88d6e37aa/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 04:56:37,837 INFO [StoreOpener-0a496ef74db8da540124a4d88d6e37aa-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:37,839 DEBUG [StoreOpener-0a496ef74db8da540124a4d88d6e37aa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C 2023-04-05 04:56:37,839 DEBUG [StoreOpener-0a496ef74db8da540124a4d88d6e37aa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C 2023-04-05 04:56:37,839 INFO [StoreOpener-0a496ef74db8da540124a4d88d6e37aa-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 04:56:37,839 INFO [StoreOpener-0a496ef74db8da540124a4d88d6e37aa-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0a496ef74db8da540124a4d88d6e37aa columnFamilyName C 2023-04-05 04:56:37,840 INFO [StoreOpener-0a496ef74db8da540124a4d88d6e37aa-1] regionserver.HStore(310): Store=0a496ef74db8da540124a4d88d6e37aa/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 04:56:37,840 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:37,841 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:37,842 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:37,845 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-05 04:56:37,847 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:37,850 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-05 04:56:37,851 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 0a496ef74db8da540124a4d88d6e37aa; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61724100, jitterRate=-0.08023923635482788}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-05 04:56:37,851 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:56:37,852 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa., pid=18, masterSystemTime=1680670597824 2023-04-05 04:56:37,854 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:37,854 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:37,855 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=0a496ef74db8da540124a4d88d6e37aa, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:37,855 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670597855"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680670597855"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680670597855"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680670597855"}]},"ts":"1680670597855"} 2023-04-05 04:56:37,859 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-05 04:56:37,859 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 in 187 msec 2023-04-05 04:56:37,861 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-05 04:56:37,861 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=0a496ef74db8da540124a4d88d6e37aa, ASSIGN in 347 msec 2023-04-05 04:56:37,862 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-05 04:56:37,862 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680670597862"}]},"ts":"1680670597862"} 2023-04-05 04:56:37,864 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-05 04:56:37,866 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-05 04:56:37,867 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1990 sec 2023-04-05 04:56:38,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-05 04:56:38,779 INFO [Listener at localhost.localdomain/34579] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-05 04:56:38,781 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x65f279d2 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:56:38,790 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@556231fa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:56:38,792 DEBUG [Listener at localhost.localdomain/34579] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:56:38,798 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39346, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:56:38,800 DEBUG [Listener at localhost.localdomain/34579] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-05 04:56:38,803 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:35914, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-05 04:56:38,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-05 04:56:38,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-05 04:56:38,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-05 04:56:39,229 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-05 04:56:39,230 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-05 04:56:39,231 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-05 04:56:39,235 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-05 04:56:39,249 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0a496ef74db8da540124a4d88d6e37aa, REOPEN/MOVE}] 2023-04-05 04:56:39,250 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0a496ef74db8da540124a4d88d6e37aa, REOPEN/MOVE 2023-04-05 04:56:39,251 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=0a496ef74db8da540124a4d88d6e37aa, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:39,251 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670599251"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680670599251"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680670599251"}]},"ts":"1680670599251"} 2023-04-05 04:56:39,252 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206}] 2023-04-05 04:56:39,405 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:39,406 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 0a496ef74db8da540124a4d88d6e37aa, disabling compactions & flushes 2023-04-05 04:56:39,407 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:39,407 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:39,407 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. after waiting 0 ms 2023-04-05 04:56:39,407 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:39,415 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-05 04:56:39,416 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:39,416 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:56:39,416 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 0a496ef74db8da540124a4d88d6e37aa to self. 2023-04-05 04:56:39,418 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:39,419 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=0a496ef74db8da540124a4d88d6e37aa, regionState=CLOSED 2023-04-05 04:56:39,420 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670599419"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680670599419"}]},"ts":"1680670599419"} 2023-04-05 04:56:39,424 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-05 04:56:39,424 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 in 169 msec 2023-04-05 04:56:39,425 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=0a496ef74db8da540124a4d88d6e37aa, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,38091,1680670569206; forceNewPlan=false, retain=true 2023-04-05 04:56:39,576 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=0a496ef74db8da540124a4d88d6e37aa, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:39,576 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670599576"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680670599576"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680670599576"}]},"ts":"1680670599576"} 2023-04-05 04:56:39,580 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206}] 2023-04-05 04:56:39,741 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:39,741 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 0a496ef74db8da540124a4d88d6e37aa, NAME => 'TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.', STARTKEY => '', ENDKEY => ''} 2023-04-05 04:56:39,742 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:39,742 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 04:56:39,742 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:39,743 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:39,749 INFO [StoreOpener-0a496ef74db8da540124a4d88d6e37aa-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:39,752 DEBUG [StoreOpener-0a496ef74db8da540124a4d88d6e37aa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A 2023-04-05 04:56:39,752 DEBUG [StoreOpener-0a496ef74db8da540124a4d88d6e37aa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A 2023-04-05 04:56:39,752 INFO [StoreOpener-0a496ef74db8da540124a4d88d6e37aa-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 04:56:39,755 INFO [StoreOpener-0a496ef74db8da540124a4d88d6e37aa-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0a496ef74db8da540124a4d88d6e37aa columnFamilyName A 2023-04-05 04:56:39,757 INFO [StoreOpener-0a496ef74db8da540124a4d88d6e37aa-1] regionserver.HStore(310): Store=0a496ef74db8da540124a4d88d6e37aa/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 04:56:39,757 INFO [StoreOpener-0a496ef74db8da540124a4d88d6e37aa-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:39,758 DEBUG [StoreOpener-0a496ef74db8da540124a4d88d6e37aa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B 2023-04-05 04:56:39,758 DEBUG [StoreOpener-0a496ef74db8da540124a4d88d6e37aa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B 2023-04-05 04:56:39,759 INFO [StoreOpener-0a496ef74db8da540124a4d88d6e37aa-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 04:56:39,759 INFO [StoreOpener-0a496ef74db8da540124a4d88d6e37aa-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0a496ef74db8da540124a4d88d6e37aa columnFamilyName B 2023-04-05 04:56:39,759 INFO [StoreOpener-0a496ef74db8da540124a4d88d6e37aa-1] regionserver.HStore(310): Store=0a496ef74db8da540124a4d88d6e37aa/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 04:56:39,760 INFO [StoreOpener-0a496ef74db8da540124a4d88d6e37aa-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:39,760 DEBUG [StoreOpener-0a496ef74db8da540124a4d88d6e37aa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C 2023-04-05 04:56:39,760 DEBUG [StoreOpener-0a496ef74db8da540124a4d88d6e37aa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C 2023-04-05 04:56:39,761 INFO [StoreOpener-0a496ef74db8da540124a4d88d6e37aa-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 04:56:39,761 INFO [StoreOpener-0a496ef74db8da540124a4d88d6e37aa-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0a496ef74db8da540124a4d88d6e37aa columnFamilyName C 2023-04-05 04:56:39,762 INFO [StoreOpener-0a496ef74db8da540124a4d88d6e37aa-1] regionserver.HStore(310): Store=0a496ef74db8da540124a4d88d6e37aa/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 04:56:39,762 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:39,763 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:39,764 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:39,766 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-05 04:56:39,767 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:39,768 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 0a496ef74db8da540124a4d88d6e37aa; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64654057, jitterRate=-0.036579474806785583}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-05 04:56:39,768 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:56:39,769 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa., pid=23, masterSystemTime=1680670599735 2023-04-05 04:56:39,770 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:39,770 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:39,770 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=0a496ef74db8da540124a4d88d6e37aa, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:39,771 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670599770"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680670599770"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680670599770"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680670599770"}]},"ts":"1680670599770"} 2023-04-05 04:56:39,773 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-05 04:56:39,773 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 in 192 msec 2023-04-05 04:56:39,775 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-05 04:56:39,775 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=0a496ef74db8da540124a4d88d6e37aa, REOPEN/MOVE in 524 msec 2023-04-05 04:56:39,778 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-05 04:56:39,778 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 542 msec 2023-04-05 04:56:39,780 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 965 msec 2023-04-05 04:56:39,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-05 04:56:39,787 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x3b75e45b to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:56:39,791 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2d781def, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:56:39,792 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x6017755a to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:56:39,796 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@595355a0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:56:39,796 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x633c81ba to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:56:39,799 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@38da9522, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:56:39,800 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x661b21c2 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:56:39,819 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a84dba6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:56:39,819 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x43a14bcf to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:56:39,827 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7c0b7748, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:56:39,828 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x2c411e08 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:56:39,838 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@bd79dff, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:56:39,839 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x7605c204 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:56:39,846 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52902392, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:56:39,847 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x16505096 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:56:39,854 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4e1da5ba, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:56:39,855 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x03e1990e to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:56:39,863 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@17a1bd04, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:56:39,871 DEBUG [hconnection-0x4ca91490-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:56:39,872 DEBUG [hconnection-0x6f5cd9e6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:56:39,872 DEBUG [hconnection-0x18549c89-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:56:39,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:39,877 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39350, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:56:39,877 DEBUG [hconnection-0x5b4da05e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:56:39,877 DEBUG [hconnection-0x18aa6410-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:56:39,877 DEBUG [hconnection-0x2d31d386-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:56:39,877 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39348, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:56:39,877 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39354, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:56:39,877 DEBUG [hconnection-0x79f6c4cb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:56:39,877 DEBUG [hconnection-0x658cce52-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:56:39,877 DEBUG [hconnection-0x4e1218c2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:56:39,880 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39372, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:56:39,881 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39368, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:56:39,881 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39386, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:56:39,882 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39398, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:56:39,889 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39406, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:56:39,889 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39422, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:56:39,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:39,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:39,891 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:39,891 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:39,891 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:39,891 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:39,898 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:39,899 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:39,900 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:39,900 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:39,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:39,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:39,903 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:39,903 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:39,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:39,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:39,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:39,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:39,905 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:39,906 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:39,906 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:39,906 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:39,906 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:39,906 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:39,907 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:39,907 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:39,907 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:39,907 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:39,908 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0a496ef74db8da540124a4d88d6e37aa 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-05 04:56:39,909 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=A 2023-04-05 04:56:39,910 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:39,910 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=B 2023-04-05 04:56:39,910 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:39,910 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=C 2023-04-05 04:56:39,910 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:39,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:39,912 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:39,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670599933, max timestamp=1680670599939Num uniques 859; 2023-04-05 04:56:39,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:56:39,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670599933, max timestamp=1680670599939Num uniques 859; 2023-04-05 04:56:39,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:56:39,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670599933, max timestamp=1680670599939Num uniques 859; 2023-04-05 04:56:39,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:56:39,967 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:39,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39398 deadline: 1680670659962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:39,968 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:39,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39406 deadline: 1680670659965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:39,968 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:39,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39372 deadline: 1680670659965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:39,968 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680670599955, max timestamp=1680670599965Num uniques 1018; 2023-04-05 04:56:39,968 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680670599955, max timestamp=1680670599965Num uniques 1018; 2023-04-05 04:56:39,969 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-05 04:56:39,969 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680670599955, max timestamp=1680670599965Num uniques 1018; 2023-04-05 04:56:39,969 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-05 04:56:39,969 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-05 04:56:39,978 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405bd4d96c135404ba0be98eeaf62e52ffa to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405bd4d96c135404ba0be98eeaf62e52ffa 2023-04-05 04:56:40,001 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:40,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39386 deadline: 1680670660000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:40,003 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:40,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39422 deadline: 1680670660002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:40,006 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=13, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/16131ce65c824d4696821e74e7bab041 2023-04-05 04:56:40,072 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:40,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39406 deadline: 1680670660070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:40,073 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:40,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39372 deadline: 1680670660070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:40,073 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:40,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39398 deadline: 1680670660070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:40,104 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:40,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39386 deadline: 1680670660102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:40,105 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:40,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39422 deadline: 1680670660105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:40,274 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:40,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39406 deadline: 1680670660274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:40,278 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:40,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39372 deadline: 1680670660275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:40,278 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:40,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39398 deadline: 1680670660276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:40,307 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:40,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39386 deadline: 1680670660306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:40,307 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:40,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39422 deadline: 1680670660307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:40,432 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/d59e02b1a0a5495aaf6d2692223b9578 2023-04-05 04:56:40,576 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:40,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39406 deadline: 1680670660576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:40,579 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:40,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39372 deadline: 1680670660579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:40,582 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:40,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39398 deadline: 1680670660581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:40,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:40,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39386 deadline: 1680670660609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:40,612 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:40,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39422 deadline: 1680670660610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:40,852 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/e4b7263161bf4f06972ebc0ab5509095 2023-04-05 04:56:40,856 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/16131ce65c824d4696821e74e7bab041 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/16131ce65c824d4696821e74e7bab041 2023-04-05 04:56:40,860 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/16131ce65c824d4696821e74e7bab041, entries=200, sequenceid=13, filesize=31.9 K 2023-04-05 04:56:40,861 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/d59e02b1a0a5495aaf6d2692223b9578 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/d59e02b1a0a5495aaf6d2692223b9578 2023-04-05 04:56:40,870 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/d59e02b1a0a5495aaf6d2692223b9578, entries=100, sequenceid=13, filesize=9.3 K 2023-04-05 04:56:40,872 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/e4b7263161bf4f06972ebc0ab5509095 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/e4b7263161bf4f06972ebc0ab5509095 2023-04-05 04:56:40,877 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/e4b7263161bf4f06972ebc0ab5509095, entries=100, sequenceid=13, filesize=9.3 K 2023-04-05 04:56:40,878 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 0a496ef74db8da540124a4d88d6e37aa in 970ms, sequenceid=13, compaction requested=false 2023-04-05 04:56:40,878 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:56:40,878 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:40,878 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:40,878 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:40,878 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:40,878 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:40,878 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:40,879 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:40,879 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:40,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:40,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:40,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:40,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:40,880 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:40,880 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:40,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:40,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:40,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:40,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:40,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:40,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7296a913[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:40,881 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:40,881 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:40,881 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:40,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:40,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:40,882 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:40,882 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:40,882 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:40,882 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:40,882 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:40,883 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:40,883 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:40,883 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:40,883 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:40,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:40,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:40,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:40,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:40,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:40,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:40,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:40,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:40,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:40,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:40,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:40,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:40,885 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:40,885 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:40,885 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:40,885 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:40,885 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:40,895 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:40,895 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:40,895 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:40,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:40,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:40,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:40,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:40,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:40,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:40,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:40,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:40,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:40,899 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:40,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:40,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:40,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:40,900 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:40,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:40,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:40,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:40,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:40,903 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:40,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:40,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:40,904 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:40,904 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:40,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:40,904 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:40,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:40,904 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:40,904 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:40,904 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:40,904 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:40,904 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:40,904 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:40,904 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:40,904 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:40,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:40,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:40,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:41,004 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:41,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:41,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:41,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:41,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:41,011 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:41,011 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:41,011 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:41,011 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:41,011 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:41,011 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:41,012 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:41,012 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:41,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:41,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:41,013 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:41,013 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:41,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:41,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:41,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:41,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:41,014 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:41,014 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:41,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:41,014 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:41,014 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:41,014 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:41,018 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:41,018 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:41,018 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:41,018 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:41,018 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0a496ef74db8da540124a4d88d6e37aa 3/3 column families, dataSize=140.89 KB heapSize=394.69 KB 2023-04-05 04:56:41,018 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=A 2023-04-05 04:56:41,018 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:56:41,018 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=B 2023-04-05 04:56:41,018 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:56:41,019 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=C 2023-04-05 04:56:41,019 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:56:41,047 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405440a5e36ba88401297676503ad457b42 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405440a5e36ba88401297676503ad457b42 2023-04-05 04:56:41,073 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=66, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/3eba1ad15e8f4642baf821813b6e098c 2023-04-05 04:56:41,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:41,086 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:41,097 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:41,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39406 deadline: 1680670661093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:41,097 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:41,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39398 deadline: 1680670661093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:41,101 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:41,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39372 deadline: 1680670661097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:41,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:41,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39386 deadline: 1680670661115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:41,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:41,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39422 deadline: 1680670661117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:41,198 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:41,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39406 deadline: 1680670661198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:41,199 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:41,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39398 deadline: 1680670661198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:41,202 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:41,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39372 deadline: 1680670661202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:41,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:41,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39406 deadline: 1680670661402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:41,404 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:41,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39398 deadline: 1680670661402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:41,404 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:41,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39372 deadline: 1680670661404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:41,500 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/c63ba82ba4e9406fafc66190f7dd50b7 2023-04-05 04:56:41,592 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-05 04:56:41,707 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:41,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39398 deadline: 1680670661705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:41,707 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:41,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39372 deadline: 1680670661705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:41,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:41,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39406 deadline: 1680670661705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:41,912 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/127aef55aed246c98c9814a5a804604d 2023-04-05 04:56:41,916 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/3eba1ad15e8f4642baf821813b6e098c as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/3eba1ad15e8f4642baf821813b6e098c 2023-04-05 04:56:41,920 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/3eba1ad15e8f4642baf821813b6e098c, entries=150, sequenceid=66, filesize=25.1 K 2023-04-05 04:56:41,921 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/c63ba82ba4e9406fafc66190f7dd50b7 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/c63ba82ba4e9406fafc66190f7dd50b7 2023-04-05 04:56:41,925 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/c63ba82ba4e9406fafc66190f7dd50b7, entries=150, sequenceid=66, filesize=11.6 K 2023-04-05 04:56:41,926 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/127aef55aed246c98c9814a5a804604d as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/127aef55aed246c98c9814a5a804604d 2023-04-05 04:56:41,930 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/127aef55aed246c98c9814a5a804604d, entries=150, sequenceid=66, filesize=11.6 K 2023-04-05 04:56:41,931 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~394.64 KB/404112, currentSize=40.25 KB/41220 for 0a496ef74db8da540124a4d88d6e37aa in 913ms, sequenceid=66, compaction requested=false 2023-04-05 04:56:41,931 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:56:41,931 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:41,931 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:41,932 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:41,932 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:41,932 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:41,932 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:41,933 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:41,933 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:41,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:41,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:41,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:41,933 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:41,933 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:41,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:41,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:41,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:41,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:41,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:41,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:41,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65f7cbe7[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:41,934 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:41,934 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:41,935 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:41,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:41,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:41,935 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:41,935 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:41,935 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:41,935 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:41,935 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:41,936 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:41,936 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:41,936 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:41,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:41,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:41,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:41,936 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:41,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:41,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:41,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:41,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:41,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:41,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:41,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:41,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:41,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:41,938 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:41,938 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:41,938 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:41,938 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:41,938 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:41,939 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:41,939 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:41,939 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:41,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:41,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:41,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:41,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:41,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:41,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:41,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:41,940 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:41,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:41,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:41,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:41,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:41,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:41,943 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:41,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:41,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:41,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:41,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:41,944 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:41,944 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:41,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:41,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:41,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:41,944 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:41,944 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:41,944 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:41,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:41,944 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:41,944 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:41,945 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:41,945 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:41,945 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:41,945 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:41,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:41,945 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:41,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:41,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:42,045 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:42,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:42,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:42,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:42,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:42,048 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:42,048 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:42,048 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:42,048 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:42,049 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,049 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:42,050 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:42,050 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:42,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:42,051 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,051 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:42,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:42,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:42,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:42,054 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:42,074 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:42,074 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:42,074 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:42,074 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:42,074 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:42,074 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:42,074 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:42,074 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:42,074 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:42,075 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0a496ef74db8da540124a4d88d6e37aa 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-05 04:56:42,075 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=A 2023-04-05 04:56:42,075 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:42,075 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=B 2023-04-05 04:56:42,075 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:42,075 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=C 2023-04-05 04:56:42,075 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:42,100 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405cc6f7b77da5248dcaf8a69e3f3dbc3a4 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405cc6f7b77da5248dcaf8a69e3f3dbc3a4 2023-04-05 04:56:42,124 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=75, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/a0bfc164fe254bc69ec4a03a2217a2b5 2023-04-05 04:56:42,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:42,125 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:42,154 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=75 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/238cbfa302c2412884b247b71efbbbbc 2023-04-05 04:56:42,171 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670602160, max timestamp=1680670602170Num uniques 859; 2023-04-05 04:56:42,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:56:42,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670602160, max timestamp=1680670602170Num uniques 859; 2023-04-05 04:56:42,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:56:42,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670602160, max timestamp=1680670602170Num uniques 859; 2023-04-05 04:56:42,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:56:42,195 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=75 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/286d8a278e7d49b7be61ea2129c4e59c 2023-04-05 04:56:42,200 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/a0bfc164fe254bc69ec4a03a2217a2b5 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/a0bfc164fe254bc69ec4a03a2217a2b5 2023-04-05 04:56:42,211 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/a0bfc164fe254bc69ec4a03a2217a2b5, entries=150, sequenceid=75, filesize=25.1 K 2023-04-05 04:56:42,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680670602194, max timestamp=1680670602209Num uniques 1018; 2023-04-05 04:56:42,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:42,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-05 04:56:42,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39372 deadline: 1680670662208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680670602194, max timestamp=1680670602209Num uniques 1018; 2023-04-05 04:56:42,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-05 04:56:42,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680670602194, max timestamp=1680670602209Num uniques 1018; 2023-04-05 04:56:42,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-05 04:56:42,212 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/238cbfa302c2412884b247b71efbbbbc as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/238cbfa302c2412884b247b71efbbbbc 2023-04-05 04:56:42,218 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/238cbfa302c2412884b247b71efbbbbc, entries=150, sequenceid=75, filesize=11.6 K 2023-04-05 04:56:42,220 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/286d8a278e7d49b7be61ea2129c4e59c as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/286d8a278e7d49b7be61ea2129c4e59c 2023-04-05 04:56:42,228 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/286d8a278e7d49b7be61ea2129c4e59c, entries=150, sequenceid=75, filesize=11.6 K 2023-04-05 04:56:42,229 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=93.93 KB/96180 for 0a496ef74db8da540124a4d88d6e37aa in 154ms, sequenceid=75, compaction requested=true 2023-04-05 04:56:42,229 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:56:42,229 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:42,229 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:42,229 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:42,229 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,229 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:42,229 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:42,230 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,230 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:42,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:42,231 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,231 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:42,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:42,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:42,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:42,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:42,232 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:42,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7607d5a3[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:42,232 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,232 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,232 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:42,232 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:42,232 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:42,232 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,233 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:42,233 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:42,233 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:42,233 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:42,234 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:42,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:42,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:42,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:42,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:42,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,235 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0a496ef74db8da540124a4d88d6e37aa 3/3 column families, dataSize=107.34 KB heapSize=299.77 KB 2023-04-05 04:56:42,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:42,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,236 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=A 2023-04-05 04:56:42,236 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:56:42,236 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=B 2023-04-05 04:56:42,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:42,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:42,236 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:56:42,236 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:42,236 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=C 2023-04-05 04:56:42,236 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:42,236 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:42,236 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:56:42,236 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:42,237 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:42,251 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,251 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:42,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,252 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:42,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:42,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:42,253 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:42,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:42,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:42,254 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:42,254 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39422 deadline: 1680670662247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:42,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,257 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:42,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39406 deadline: 1680670662252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,257 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:42,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39386 deadline: 1680670662253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,258 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:42,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39398 deadline: 1680670662254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,260 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:42,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:42,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:42,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:42,260 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:42,260 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:42,260 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,260 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:42,260 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:42,260 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,260 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,260 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,260 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:42,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:42,260 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:42,260 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:42,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:42,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:42,264 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040591a2f7958d1b42cd9cbeed43f3b134a9 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040591a2f7958d1b42cd9cbeed43f3b134a9 2023-04-05 04:56:42,360 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:42,360 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:42,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39422 deadline: 1680670662355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:42,362 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:42,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39406 deadline: 1680670662358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,362 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:42,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39386 deadline: 1680670662358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,362 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:42,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39398 deadline: 1680670662359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:42,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:42,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:42,364 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:42,365 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:42,365 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:42,365 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:42,365 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,365 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:42,366 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:42,366 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:42,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:42,366 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:42,366 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:42,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:42,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:42,367 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:42,367 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:42,367 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:42,367 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:42,367 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:42,367 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:42,367 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:42,368 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:42,368 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:42,368 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:42,368 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:42,368 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:42,368 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:42,368 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:42,368 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,368 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:42,368 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:42,369 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,369 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:42,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:42,369 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,369 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:42,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:42,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:42,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:42,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:42,371 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:42,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7390cf75[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:42,371 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,372 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,372 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:42,372 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:42,372 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:42,372 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,372 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:42,373 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,373 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:42,373 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:42,373 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:42,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:42,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:42,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:42,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:42,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:42,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:42,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:42,376 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:42,376 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:42,376 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:42,376 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:42,376 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:42,377 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:42,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:42,378 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,378 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:42,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:42,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:42,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:42,378 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:42,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,379 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:42,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,381 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,381 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:42,381 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:42,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:42,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:42,381 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:42,381 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:42,381 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:42,381 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:42,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:42,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:42,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:42,382 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,382 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,382 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,382 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:42,382 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:42,482 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:42,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:42,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:42,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:42,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:42,486 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:42,486 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:42,486 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:42,486 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:42,486 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,486 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:42,487 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:42,487 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:42,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:42,487 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,487 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:42,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:42,488 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:42,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:42,488 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:42,488 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:42,488 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:42,488 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:42,488 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:42,488 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:42,489 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:42,489 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:42,489 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:42,489 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:42,489 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:42,489 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:42,489 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:42,489 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:42,489 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,489 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:42,489 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:42,490 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,490 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:42,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:42,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:42,490 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,490 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:42,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:42,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:42,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:42,491 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:42,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34892e25[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:42,491 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,492 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,492 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:42,492 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:42,492 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:42,492 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,492 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:42,493 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,493 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:42,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:42,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:42,493 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:42,493 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:42,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:42,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:42,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:42,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:42,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:42,495 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:42,495 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:42,495 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:42,495 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:42,495 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:42,496 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,496 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:42,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:42,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,496 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:42,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:42,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:42,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:42,497 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:42,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,498 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:42,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,501 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:42,501 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:42,502 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:42,501 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:42,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:42,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:42,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:42,502 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:42,502 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:42,502 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,502 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,502 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:42,502 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,502 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:42,502 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:42,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:42,562 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:42,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39422 deadline: 1680670662562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,565 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:42,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39406 deadline: 1680670662563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,567 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:42,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39386 deadline: 1680670662564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,567 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:42,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39398 deadline: 1680670662565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,602 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:42,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:42,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:42,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:42,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:42,610 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:42,610 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:42,610 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:42,610 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:42,611 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,611 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:42,611 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:42,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:42,612 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:42,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:42,612 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,612 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:42,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:42,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:42,613 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:42,613 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:42,613 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:42,613 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:42,613 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:42,613 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:42,613 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:42,613 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:42,613 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:42,613 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:42,613 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:42,613 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:42,613 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:42,614 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:42,614 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,614 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:42,614 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:42,615 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,615 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:42,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:42,615 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,615 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:42,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:42,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:42,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:42,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:42,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ffa63a0[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:42,618 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:42,618 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,627 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,627 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:42,627 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:42,627 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:42,627 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,627 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:42,628 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,628 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:42,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:42,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:42,628 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:42,629 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:42,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:42,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:42,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:42,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:42,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:42,631 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:42,631 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:42,631 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:42,631 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:42,631 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:42,632 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,632 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,632 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:42,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:42,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:42,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,633 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:42,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:42,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:42,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:42,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,634 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:42,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,635 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:42,636 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:42,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:42,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:42,636 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,636 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:42,636 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:42,636 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,636 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,636 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:42,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:42,636 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,636 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:42,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:42,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:42,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:42,636 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:42,636 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:42,676 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=126, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/2231dfbcfb40404c80e8afcb4efe91b1 2023-04-05 04:56:42,737 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:42,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:42,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:42,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:42,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:42,740 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:42,740 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:42,741 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:42,741 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:42,741 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,741 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:42,742 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,742 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:42,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:42,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:42,742 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,742 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:42,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:42,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:42,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:42,743 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:42,743 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:42,743 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:42,743 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:42,743 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:42,743 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:42,743 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:42,743 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:42,743 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:42,743 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:42,743 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:42,743 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:42,744 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:42,744 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:42,744 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,744 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:42,744 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:42,744 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,744 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:42,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:42,745 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,745 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:42,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:42,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:42,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:42,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:42,746 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:42,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27980931[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:42,746 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,746 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,747 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:42,747 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:42,747 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,747 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:42,747 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:42,747 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,747 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:42,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:42,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:42,747 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:42,748 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:42,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:42,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:42,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:42,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:42,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:42,749 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:42,749 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:42,749 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:42,749 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:42,749 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:42,750 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:42,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:42,750 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,750 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:42,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:42,751 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:42,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:42,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:42,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,752 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:42,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,753 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,753 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:42,753 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:42,753 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:42,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:42,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:42,753 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,753 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:42,753 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:42,753 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:42,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:42,753 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:42,753 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:42,753 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:42,753 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:42,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:42,854 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:42,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:42,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:42,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:42,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:42,856 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:42,856 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:42,856 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:42,857 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:42,857 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,857 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:42,858 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:42,858 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:42,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:42,858 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,858 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:42,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:42,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:42,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:42,858 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:42,859 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:42,859 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:42,859 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:42,859 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:42,859 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:42,859 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:42,859 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:42,859 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:42,859 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:42,859 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:42,859 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:42,859 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:42,859 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:42,859 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,859 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:42,859 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:42,860 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,860 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:42,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:42,860 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,860 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:42,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:42,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:42,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:42,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:42,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@528691d0[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:42,861 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:42,862 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,862 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,862 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,862 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:42,862 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:42,862 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:42,862 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:42,863 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:42,863 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:42,863 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,863 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:42,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:42,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:42,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:42,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:42,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:42,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:42,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:42,866 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:42,866 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:42,866 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:42,866 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:42,866 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:42,867 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,867 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,867 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:42,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:42,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:42,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:42,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39422 deadline: 1680670662865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:42,867 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:42,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:42,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:42,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,869 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:42,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:42,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39386 deadline: 1680670662868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:42,870 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:42,870 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39406 deadline: 1680670662869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,871 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:42,871 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:42,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:42,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:42,871 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:42,871 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:42,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:42,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:42,870 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:42,871 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:42,871 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,871 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,871 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,871 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:42,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:42,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:42,873 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:42,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39398 deadline: 1680670662870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,971 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:42,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:42,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:42,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:42,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:42,979 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:42,979 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:42,979 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:42,979 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:42,980 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,980 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:42,981 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,981 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:42,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:42,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:42,981 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,981 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:42,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:42,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:42,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:42,982 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:42,982 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:42,982 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:42,982 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:42,982 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:42,982 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:42,986 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:42,986 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:42,986 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:42,986 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:42,986 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:42,986 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:42,986 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:42,986 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:42,986 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,986 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:42,986 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:42,987 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,987 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:42,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:42,987 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,988 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:42,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:42,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:42,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:42,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:42,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f8365a3[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:42,989 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:42,989 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,990 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,990 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:42,990 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:42,990 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:42,990 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,990 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:42,991 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,991 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:42,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:42,991 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:42,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:42,991 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:42,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:42,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:42,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:42,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:42,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:42,993 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:42,993 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:42,993 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:42,993 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:42,993 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:42,994 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,994 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,994 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:42,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:42,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:42,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:42,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:42,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:42,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,996 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:42,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:42,997 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:42,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,999 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,999 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:42,999 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:42,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:42,999 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:42,999 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:42,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:42,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:42,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:42,999 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:42,999 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:42,999 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,999 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:42,999 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:42,999 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:42,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:43,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:42,999 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:43,087 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=126 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/43a7d26cbd744889bbc61859eeec8003 2023-04-05 04:56:43,100 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:43,100 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=126 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/07cecf3d8bb7404eac56dca98661dc9a 2023-04-05 04:56:43,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:43,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:43,105 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/2231dfbcfb40404c80e8afcb4efe91b1 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/2231dfbcfb40404c80e8afcb4efe91b1 2023-04-05 04:56:43,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:43,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:43,106 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:43,107 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:43,107 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:43,107 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:43,107 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,107 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:43,108 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:43,108 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:43,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:43,108 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,108 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:43,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:43,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:43,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:43,109 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:43,109 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:43,109 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:43,109 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:43,109 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:43,109 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:43,109 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:43,109 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:43,109 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:43,109 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:43,109 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/2231dfbcfb40404c80e8afcb4efe91b1, entries=150, sequenceid=126, filesize=25.1 K 2023-04-05 04:56:43,109 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:43,109 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:43,109 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:43,109 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:43,109 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,110 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:43,110 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:43,110 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/43a7d26cbd744889bbc61859eeec8003 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/43a7d26cbd744889bbc61859eeec8003 2023-04-05 04:56:43,111 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,111 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:43,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:43,111 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,111 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:43,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:43,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:43,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:43,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:43,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13d8f490[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:43,113 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:43,113 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,113 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,114 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:43,114 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:43,114 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:43,114 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,114 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:43,114 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/43a7d26cbd744889bbc61859eeec8003, entries=150, sequenceid=126, filesize=11.6 K 2023-04-05 04:56:43,115 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:43,115 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:43,115 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,115 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:43,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:43,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:43,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:43,115 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/07cecf3d8bb7404eac56dca98661dc9a as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/07cecf3d8bb7404eac56dca98661dc9a 2023-04-05 04:56:43,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:43,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:43,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:43,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:43,117 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:43,117 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:43,117 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:43,117 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:43,117 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:43,118 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,118 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:43,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:43,118 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:43,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:43,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:43,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:43,119 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:43,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,121 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:43,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,122 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/07cecf3d8bb7404eac56dca98661dc9a, entries=150, sequenceid=126, filesize=11.6 K 2023-04-05 04:56:43,123 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,123 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~339.56 KB/347712, currentSize=67.09 KB/68700 for 0a496ef74db8da540124a4d88d6e37aa in 889ms, sequenceid=126, compaction requested=true 2023-04-05 04:56:43,123 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:43,123 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:43,123 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:43,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:43,123 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:43,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:43,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:43,123 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:56:43,123 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:43,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:43,123 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:56:43,123 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,123 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,123 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:43,123 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:43,123 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:56:43,123 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:56:43,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:43,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:43,123 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:56:43,123 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:56:43,123 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,124 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,125 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 109920 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:56:43,125 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): 0a496ef74db8da540124a4d88d6e37aa/A is initiating minor compaction (all files) 2023-04-05 04:56:43,125 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:56:43,125 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0a496ef74db8da540124a4d88d6e37aa/A in TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:43,125 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): 0a496ef74db8da540124a4d88d6e37aa/B is initiating minor compaction (all files) 2023-04-05 04:56:43,125 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0a496ef74db8da540124a4d88d6e37aa/B in TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:43,125 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/16131ce65c824d4696821e74e7bab041, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/3eba1ad15e8f4642baf821813b6e098c, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/a0bfc164fe254bc69ec4a03a2217a2b5, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/2231dfbcfb40404c80e8afcb4efe91b1] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp, totalSize=107.3 K 2023-04-05 04:56:43,125 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/d59e02b1a0a5495aaf6d2692223b9578, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/c63ba82ba4e9406fafc66190f7dd50b7, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/238cbfa302c2412884b247b71efbbbbc, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/43a7d26cbd744889bbc61859eeec8003] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp, totalSize=44.1 K 2023-04-05 04:56:43,125 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 16131ce65c824d4696821e74e7bab041, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680670599903 2023-04-05 04:56:43,125 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting d59e02b1a0a5495aaf6d2692223b9578, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680670599907 2023-04-05 04:56:43,126 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 3eba1ad15e8f4642baf821813b6e098c, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1680670599987 2023-04-05 04:56:43,126 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting c63ba82ba4e9406fafc66190f7dd50b7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1680670599987 2023-04-05 04:56:43,126 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting a0bfc164fe254bc69ec4a03a2217a2b5, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1680670601089 2023-04-05 04:56:43,126 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 238cbfa302c2412884b247b71efbbbbc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1680670601089 2023-04-05 04:56:43,126 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 2231dfbcfb40404c80e8afcb4efe91b1, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1680670602233 2023-04-05 04:56:43,126 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 43a7d26cbd744889bbc61859eeec8003, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1680670602233 2023-04-05 04:56:43,136 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): 0a496ef74db8da540124a4d88d6e37aa#B#compaction#125 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:56:43,155 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0a496ef74db8da540124a4d88d6e37aa#A#compaction#126 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:56:43,160 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/9b0cf2a048f14847ba39afe51d58b0d7 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/9b0cf2a048f14847ba39afe51d58b0d7 2023-04-05 04:56:43,170 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0a496ef74db8da540124a4d88d6e37aa/B of 0a496ef74db8da540124a4d88d6e37aa into 9b0cf2a048f14847ba39afe51d58b0d7(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:56:43,170 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:56:43,170 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa., storeName=0a496ef74db8da540124a4d88d6e37aa/B, priority=12, startTime=1680670603123; duration=0sec 2023-04-05 04:56:43,170 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:56:43,170 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:56:43,171 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:56:43,171 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): 0a496ef74db8da540124a4d88d6e37aa/C is initiating minor compaction (all files) 2023-04-05 04:56:43,172 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0a496ef74db8da540124a4d88d6e37aa/C in TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:43,172 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/e4b7263161bf4f06972ebc0ab5509095, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/127aef55aed246c98c9814a5a804604d, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/286d8a278e7d49b7be61ea2129c4e59c, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/07cecf3d8bb7404eac56dca98661dc9a] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp, totalSize=44.1 K 2023-04-05 04:56:43,172 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting e4b7263161bf4f06972ebc0ab5509095, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680670599907 2023-04-05 04:56:43,172 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 127aef55aed246c98c9814a5a804604d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1680670599987 2023-04-05 04:56:43,173 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 286d8a278e7d49b7be61ea2129c4e59c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1680670601089 2023-04-05 04:56:43,173 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 07cecf3d8bb7404eac56dca98661dc9a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1680670602233 2023-04-05 04:56:43,187 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): 0a496ef74db8da540124a4d88d6e37aa#C#compaction#127 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:56:43,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:43,217 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0a496ef74db8da540124a4d88d6e37aa 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-05 04:56:43,217 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=A 2023-04-05 04:56:43,217 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:43,217 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=B 2023-04-05 04:56:43,217 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:43,217 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=C 2023-04-05 04:56:43,217 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:43,223 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:43,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:43,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:43,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:43,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:43,227 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:43,227 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:43,227 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:43,227 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:43,227 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,227 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:43,228 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,228 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:43,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:43,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:43,228 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,228 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:43,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:43,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:43,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:43,228 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:43,229 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:43,229 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:43,229 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:43,229 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:43,229 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:43,229 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:43,229 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:43,229 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:43,229 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:43,229 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:43,229 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:43,229 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:43,229 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:43,229 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,229 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:43,229 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:43,230 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,230 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:43,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:43,231 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,231 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:43,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:43,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:43,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:43,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:43,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@746ea65b[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:43,232 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:43,232 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,232 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,232 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:43,232 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:43,232 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:43,233 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,233 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:43,233 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,233 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:43,233 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:43,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:43,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:43,233 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:43,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:43,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:43,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:43,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:43,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:43,236 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:43,236 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:43,236 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:43,236 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:43,236 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:43,237 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:43,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:43,237 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,237 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:43,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:43,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:43,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:43,238 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:43,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,239 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:43,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,240 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,240 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:43,240 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:43,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:43,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:43,241 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:43,241 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,241 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,241 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,241 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,241 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:43,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:43,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:43,242 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:43,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:43,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:43,242 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:43,242 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:43,274 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:43,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39372 deadline: 1680670663270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,342 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:43,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:43,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:43,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:43,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:43,346 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:43,346 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:43,346 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:43,346 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:43,346 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,346 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:43,347 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:43,347 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:43,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:43,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:43,347 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,347 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:43,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:43,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:43,348 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:43,348 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:43,348 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:43,348 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:43,348 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:43,348 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:43,349 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:43,349 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:43,349 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:43,349 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:43,349 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:43,349 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:43,349 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:43,349 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:43,349 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,349 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:43,349 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:43,350 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,350 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:43,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:43,351 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,351 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:43,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:43,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:43,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:43,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:43,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1200b117[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:43,354 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:43,354 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,355 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,355 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:43,355 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:43,355 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:43,355 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,355 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:43,357 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,357 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:43,357 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:43,357 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:43,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:43,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:43,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:43,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:43,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:43,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:43,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:43,360 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:43,360 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:43,360 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:43,360 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:43,360 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:43,361 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,361 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:43,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:43,361 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:43,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:43,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:43,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:43,362 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:43,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,363 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:43,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,369 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,369 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:43,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:43,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:43,369 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:43,369 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,369 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,369 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,369 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,369 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:43,369 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:43,369 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:43,369 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:43,370 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:43,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:43,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:43,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:43,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:43,372 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:43,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39422 deadline: 1680670663368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,375 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:43,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39386 deadline: 1680670663374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:43,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39372 deadline: 1680670663375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,376 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:43,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39406 deadline: 1680670663376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,379 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:43,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39398 deadline: 1680670663377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,470 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:43,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:43,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:43,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:43,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:43,472 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:43,472 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:43,472 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:43,472 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:43,473 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,473 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:43,473 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,473 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:43,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:43,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:43,474 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,474 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:43,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:43,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:43,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:43,474 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:43,476 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:43,476 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:43,476 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:43,476 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:43,476 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:43,476 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:43,476 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:43,476 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:43,476 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:43,476 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:43,476 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:43,476 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:43,476 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:43,476 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,476 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:43,476 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:43,478 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,478 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:43,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:43,478 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,478 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:43,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:43,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:43,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:43,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:43,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@261a6089[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:43,479 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:43,479 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,480 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,480 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:43,480 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:43,480 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:43,480 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,480 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:43,480 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,480 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:43,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:43,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:43,481 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:43,481 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:43,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:43,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:43,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:43,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:43,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:43,482 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:43,482 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:43,482 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:43,482 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:43,483 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:43,483 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,483 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:43,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:43,483 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:43,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,484 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:43,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:43,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:43,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:43,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,485 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:43,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,486 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,487 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:43,487 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:43,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:43,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:43,487 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:43,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:43,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:43,487 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,487 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:43,487 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:43,487 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:43,487 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:43,487 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:43,487 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,487 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:43,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:43,578 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:43,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39372 deadline: 1680670663577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,587 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:43,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:43,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:43,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:43,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:43,589 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:43,590 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:43,590 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:43,590 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:43,590 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,590 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:43,590 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,590 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:43,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:43,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:43,591 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,591 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:43,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:43,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:43,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:43,591 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:43,591 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:43,592 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:43,592 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:43,592 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:43,592 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:43,592 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:43,592 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:43,592 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:43,592 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:43,592 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:43,592 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:43,592 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:43,592 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:43,592 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,592 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:43,592 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:43,593 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,593 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:43,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:43,593 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,593 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:43,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:43,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:43,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:43,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:43,594 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:43,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72b27cae[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:43,594 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,595 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,595 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:43,595 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:43,595 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:43,595 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:43,595 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:43,596 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:43,602 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,602 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:43,602 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:43,602 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:43,602 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:43,603 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:43,603 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:43,603 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:43,603 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/077e1e0832f84261b1b78b6990e361ba as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/077e1e0832f84261b1b78b6990e361ba 2023-04-05 04:56:43,609 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0a496ef74db8da540124a4d88d6e37aa/C of 0a496ef74db8da540124a4d88d6e37aa into 077e1e0832f84261b1b78b6990e361ba(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:56:43,609 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:56:43,609 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa., storeName=0a496ef74db8da540124a4d88d6e37aa/C, priority=12, startTime=1680670603123; duration=0sec 2023-04-05 04:56:43,609 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:56:43,612 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,612 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,612 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:43,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:43,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:43,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:43,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:43,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:43,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,613 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/46255102075e4d8a82f802be2ab91ddf as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/46255102075e4d8a82f802be2ab91ddf 2023-04-05 04:56:43,613 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:43,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,615 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:43,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,621 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0a496ef74db8da540124a4d88d6e37aa/A of 0a496ef74db8da540124a4d88d6e37aa into 46255102075e4d8a82f802be2ab91ddf(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:56:43,621 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:56:43,621 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa., storeName=0a496ef74db8da540124a4d88d6e37aa/A, priority=12, startTime=1680670603123; duration=0sec 2023-04-05 04:56:43,621 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:56:43,621 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,621 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:43,621 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:43,621 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,622 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,622 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,622 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:43,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:43,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:43,621 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:43,622 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:43,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:43,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:43,622 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,622 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:43,622 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:43,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:43,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:43,630 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040587ffdfc611374dee8a4f61597dfe445b to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040587ffdfc611374dee8a4f61597dfe445b 2023-04-05 04:56:43,722 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:43,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:43,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:43,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:43,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:43,725 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:43,725 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:43,725 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:43,725 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:43,726 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,726 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:43,726 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,726 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:43,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:43,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:43,726 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,726 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:43,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:43,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:43,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:43,727 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:43,727 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:43,727 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:43,727 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:43,728 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:43,728 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:43,728 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:43,728 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:43,728 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:43,728 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:43,728 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:43,728 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:43,728 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:43,728 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:43,728 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,728 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:43,728 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:43,729 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,729 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:43,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:43,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:43,729 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,730 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:43,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:43,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:43,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:43,731 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:43,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73ef50dd[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:43,731 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,731 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,731 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:43,731 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:43,732 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:43,733 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,733 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:43,734 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,734 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:43,734 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:43,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:43,734 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:43,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:43,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:43,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:43,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:43,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:43,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:43,737 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:43,737 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:43,737 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:43,737 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:43,737 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:43,743 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,743 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,743 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:43,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:43,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:43,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:43,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:43,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:43,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,744 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:43,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,745 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:43,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,754 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,754 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:43,754 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:43,754 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:43,754 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:43,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:43,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:43,754 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:43,754 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,756 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,756 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,756 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:43,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:43,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:43,756 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:43,756 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:43,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:43,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:43,856 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:43,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:43,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:43,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:43,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:43,860 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:43,860 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:43,860 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:43,860 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:43,861 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,861 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:43,861 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:43,861 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:43,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:43,862 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,862 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:43,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:43,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:43,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:43,862 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:43,862 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:43,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:43,863 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:43,863 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:43,863 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:43,863 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:43,863 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:43,863 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:43,863 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:43,863 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:43,863 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:43,863 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:43,863 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:43,863 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,863 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:43,863 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:43,864 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,864 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:43,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:43,864 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,864 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:43,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:43,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:43,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:43,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:43,866 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:43,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6470dfbe[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:43,866 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,866 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,866 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:43,866 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:43,866 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:43,867 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,867 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:43,867 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,867 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:43,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:43,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:43,867 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:43,867 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:43,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:43,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:43,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:43,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:43,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:43,871 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:43,871 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:43,871 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:43,872 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:43,872 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:43,872 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,872 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,872 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:43,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:43,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:43,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:43,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:43,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:43,873 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:43,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,882 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:43,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:43,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:43,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39372 deadline: 1680670663881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,892 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,892 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:43,892 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:43,892 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,892 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,892 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:43,892 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:43,892 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:43,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:43,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:43,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:43,892 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:43,892 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:43,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:43,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:43,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:43,894 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:43,894 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:43,994 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:43,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:43,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:43,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:43,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:43,997 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:43,997 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:43,998 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:43,998 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:43,998 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:43,998 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:43,999 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:43,999 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:43,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:43,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:43,999 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:43,999 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:43,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:44,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:44,000 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:44,000 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:44,000 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:44,000 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:44,000 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:44,000 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:44,000 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:44,000 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:44,000 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:44,000 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:44,000 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:44,001 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:44,001 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:44,001 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:44,001 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,001 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:44,001 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:44,002 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,002 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:44,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:44,002 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,002 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:44,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:44,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:44,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:44,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:44,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:44,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@111be591[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:44,004 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:44,004 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,004 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,005 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:44,005 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,005 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:44,005 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:44,005 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:44,006 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:44,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:44,006 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:44,006 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:44,006 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:44,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:44,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:44,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:44,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:44,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:44,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:44,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:44,008 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:44,008 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:44,008 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:44,008 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:44,008 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:44,010 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,010 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:44,010 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:44,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:44,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:44,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:44,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:44,011 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:44,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:44,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:44,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:44,013 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:44,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,016 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,016 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:44,016 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:44,017 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,017 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,017 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,017 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,017 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:44,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:44,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:44,017 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:44,017 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:44,017 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:44,017 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:44,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:44,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:44,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:44,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:44,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:44,041 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=141, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/6565982749d040e69836b3fc37c0ad05 2023-04-05 04:56:44,117 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:44,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:44,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:44,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:44,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:44,120 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:44,120 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:44,120 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:44,120 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:44,120 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,120 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:44,121 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:44,121 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:44,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:44,121 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,122 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:44,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:44,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:44,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:44,122 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:44,122 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:44,122 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:44,122 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:44,122 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:44,123 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:44,123 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:44,123 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:44,123 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:44,123 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:44,123 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:44,123 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:44,123 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:44,123 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:44,123 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,123 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:44,123 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:44,124 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,124 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:44,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:44,124 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,124 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:44,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:44,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:44,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:44,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:44,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:44,126 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:44,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70adbc10[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:44,126 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,127 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,127 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:44,127 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:44,127 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:44,127 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,127 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:44,128 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:44,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:44,128 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:44,128 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:44,128 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:44,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:44,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:44,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:44,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:44,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:44,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:44,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:44,130 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:44,130 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:44,130 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:44,130 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:44,130 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:44,131 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,131 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:44,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:44,131 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:44,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:44,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:44,132 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:44,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:44,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:44,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:44,133 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:44,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:44,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,135 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,135 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:44,135 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:44,135 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:44,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:44,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:44,135 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:44,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:44,135 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:44,135 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:44,135 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:44,135 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:44,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:44,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:44,135 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,136 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,135 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:44,236 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:44,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:44,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:44,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:44,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:44,239 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:44,239 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:44,239 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:44,239 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:44,239 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,239 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:44,240 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:44,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:44,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:44,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:44,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:44,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:44,241 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:44,241 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:44,242 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:44,242 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:44,242 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:44,242 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,242 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:44,242 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:44,242 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:44,242 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:44,242 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,242 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:44,242 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:44,242 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:44,242 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:44,242 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:44,242 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:44,242 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,242 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:44,242 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:44,243 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,243 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:44,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:44,243 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,243 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:44,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:44,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:44,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:44,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:44,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:44,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79cd7360[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:44,245 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:44,245 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,245 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,245 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:44,245 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:44,246 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:44,246 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,246 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:44,246 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:44,246 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,246 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:44,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:44,246 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:44,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:44,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:44,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:44,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:44,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:44,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:44,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:44,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:44,248 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:44,248 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:44,248 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:44,248 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:44,248 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:44,251 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,251 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,251 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:44,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:44,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:44,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:44,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:44,251 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:44,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:44,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:44,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:44,252 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:44,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:44,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,254 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,254 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:44,254 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:44,254 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,254 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:44,254 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,254 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:44,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:44,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:44,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:44,254 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:44,254 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,255 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,255 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:44,255 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:44,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:44,255 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:44,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:44,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:44,355 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:44,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:44,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:44,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:44,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:44,358 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:44,358 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:44,358 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:44,358 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:44,358 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,358 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:44,359 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:44,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:44,359 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:44,359 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,359 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:44,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:44,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:44,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:44,360 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:44,360 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:44,360 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:44,360 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:44,360 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:44,360 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:44,360 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:44,360 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:44,360 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:44,360 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:44,360 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:44,360 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:44,361 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:44,361 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:44,361 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,361 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:44,361 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:44,361 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,362 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:44,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:44,362 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,362 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:44,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:44,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:44,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:44,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:44,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:44,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ba2cec8[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:44,363 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:44,363 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,364 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,364 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:44,364 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:44,364 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:44,364 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,364 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:44,365 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,365 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:44,365 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:44,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:44,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:44,365 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:44,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:44,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:44,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:44,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:44,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:44,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:44,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:44,367 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:44,367 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:44,367 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:44,367 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:44,367 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:44,368 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,368 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,368 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:44,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:44,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:44,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:44,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:44,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:44,369 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:44,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:44,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:44,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,370 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:44,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:44,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,371 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,371 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:44,371 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:44,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:44,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:44,371 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:44,372 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,372 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,372 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,371 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:44,372 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:44,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:44,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:44,372 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:44,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:44,372 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:44,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:44,372 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:44,377 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:44,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39406 deadline: 1680670664377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,382 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:44,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39422 deadline: 1680670664382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,383 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:44,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39386 deadline: 1680670664382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,386 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:44,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39398 deadline: 1680670664386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,389 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:44,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39372 deadline: 1680670664388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,453 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/f191857faca74ddc8baae6cf23c40e89 2023-04-05 04:56:44,464 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/19c08e36c73444e58a2684a11148ed88 2023-04-05 04:56:44,469 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/6565982749d040e69836b3fc37c0ad05 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/6565982749d040e69836b3fc37c0ad05 2023-04-05 04:56:44,472 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:44,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:44,474 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/6565982749d040e69836b3fc37c0ad05, entries=200, sequenceid=141, filesize=32.1 K 2023-04-05 04:56:44,475 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/f191857faca74ddc8baae6cf23c40e89 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/f191857faca74ddc8baae6cf23c40e89 2023-04-05 04:56:44,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:44,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:44,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:44,479 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:44,479 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:44,479 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:44,479 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:44,479 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,479 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:44,480 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:44,480 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:44,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:44,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:44,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,481 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,481 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:44,482 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/f191857faca74ddc8baae6cf23c40e89, entries=150, sequenceid=141, filesize=11.8 K 2023-04-05 04:56:44,483 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/19c08e36c73444e58a2684a11148ed88 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/19c08e36c73444e58a2684a11148ed88 2023-04-05 04:56:44,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:44,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:44,483 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:44,484 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:44,484 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:44,484 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:44,484 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:44,484 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:44,486 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:44,486 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:44,486 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:44,486 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:44,486 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:44,486 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:44,486 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:44,486 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:44,486 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,486 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:44,486 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:44,488 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:44,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:44,488 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,488 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/19c08e36c73444e58a2684a11148ed88, entries=150, sequenceid=141, filesize=11.8 K 2023-04-05 04:56:44,488 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,488 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:44,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:44,489 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 0a496ef74db8da540124a4d88d6e37aa in 1273ms, sequenceid=141, compaction requested=false 2023-04-05 04:56:44,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:44,489 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:56:44,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:44,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:44,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:44,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@555b9740[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:44,490 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:44,490 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,491 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,491 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:44,491 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:44,491 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:44,491 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,491 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:44,492 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:44,492 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,492 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:44,492 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:44,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:44,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:44,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:44,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:44,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:44,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:44,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:44,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:44,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:44,494 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:44,494 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:44,494 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:44,494 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:44,494 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:44,495 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,495 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:44,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:44,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:44,496 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:44,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:44,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:44,496 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:44,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:44,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:44,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,498 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:44,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:44,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,524 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,524 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:44,524 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:44,524 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,524 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:44,524 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:44,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:44,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:44,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:44,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:44,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:44,525 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:44,524 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:44,524 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,525 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:44,525 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:44,525 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:44,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:44,625 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:44,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:44,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:44,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:44,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:44,627 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:44,627 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:44,627 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:44,627 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:44,628 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,628 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:44,628 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:44,628 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:44,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:44,629 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:44,629 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:44,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:44,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:44,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:44,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:44,629 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:44,629 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:44,629 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:44,630 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:44,630 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:44,630 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:44,630 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:44,630 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:44,630 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:44,630 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0a496ef74db8da540124a4d88d6e37aa 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-05 04:56:44,630 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=A 2023-04-05 04:56:44,630 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:44,630 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=B 2023-04-05 04:56:44,630 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:44,630 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=C 2023-04-05 04:56:44,630 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:45,062 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405ca71227a2bee405697cbe0015e110692 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405ca71227a2bee405697cbe0015e110692 2023-04-05 04:56:45,076 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=161, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/2604088d799a4626a5262d47fcca7a83 2023-04-05 04:56:45,099 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=161 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/78a13d97b8d840bca3fbc69c706c30e6 2023-04-05 04:56:45,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:45,399 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:45,425 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:45,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39372 deadline: 1680670665424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:45,516 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=161 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/89cc91b1a22a4b11895f83b5b00a5eb7 2023-04-05 04:56:45,520 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/2604088d799a4626a5262d47fcca7a83 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/2604088d799a4626a5262d47fcca7a83 2023-04-05 04:56:45,525 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/2604088d799a4626a5262d47fcca7a83, entries=150, sequenceid=161, filesize=25.3 K 2023-04-05 04:56:45,526 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/78a13d97b8d840bca3fbc69c706c30e6 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/78a13d97b8d840bca3fbc69c706c30e6 2023-04-05 04:56:45,529 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:45,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39372 deadline: 1680670665527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:45,531 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/78a13d97b8d840bca3fbc69c706c30e6, entries=150, sequenceid=161, filesize=11.8 K 2023-04-05 04:56:45,532 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/89cc91b1a22a4b11895f83b5b00a5eb7 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/89cc91b1a22a4b11895f83b5b00a5eb7 2023-04-05 04:56:45,535 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/89cc91b1a22a4b11895f83b5b00a5eb7, entries=150, sequenceid=161, filesize=11.8 K 2023-04-05 04:56:45,536 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 0a496ef74db8da540124a4d88d6e37aa in 906ms, sequenceid=161, compaction requested=true 2023-04-05 04:56:45,536 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:56:45,537 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:45,537 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:45,537 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:45,537 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:45,537 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:45,537 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:45,538 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:45,538 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:45,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:45,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:45,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:45,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:45,539 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:45,539 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:45,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:45,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:45,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:45,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:45,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:45,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@488bfcc7[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:45,540 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:45,540 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:45,541 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:45,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:45,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:45,541 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:45,541 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:45,541 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:45,541 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:45,541 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:45,541 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:45,541 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:45,541 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:45,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:45,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:45,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:45,542 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:45,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:45,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:45,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:45,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:45,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:45,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:45,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:45,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:45,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:45,543 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:45,544 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:45,544 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:45,544 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:45,544 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:45,544 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:45,544 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:45,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:45,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:45,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:45,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:45,545 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:45,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:45,545 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:45,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:45,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:45,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:45,546 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:45,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:45,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:45,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:45,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:45,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:45,547 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:45,547 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:45,548 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:45,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:45,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:45,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:45,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:45,548 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:45,548 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:45,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:45,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:45,548 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:45,548 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:45,548 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:45,548 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:45,548 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:45,548 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:45,548 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:45,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:45,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:45,648 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:45,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:45,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:45,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:45,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:45,651 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:45,651 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:45,651 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:45,651 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:45,651 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:45,651 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:45,652 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:45,652 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:45,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:45,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:45,652 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:45,652 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:45,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:45,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:45,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:45,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:45,653 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:45,653 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:45,653 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:45,653 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:45,653 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:45,653 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:45,653 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:45,653 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:45,653 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:45,653 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:45,653 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0a496ef74db8da540124a4d88d6e37aa 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-05 04:56:45,654 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=A 2023-04-05 04:56:45,654 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:45,654 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=B 2023-04-05 04:56:45,654 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:45,654 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=C 2023-04-05 04:56:45,654 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:45,671 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405ee7383dc59974a94b0e8eec9c821a39a to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405ee7383dc59974a94b0e8eec9c821a39a 2023-04-05 04:56:45,675 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=175, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/cb3d50f46248404bac70b1229f52c270 2023-04-05 04:56:45,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:45,734 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:45,785 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:45,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39372 deadline: 1680670665783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:45,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:45,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39372 deadline: 1680670665886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:46,085 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=175 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/d2d9e490fdb24f1abd1f1e836db84078 2023-04-05 04:56:46,091 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:46,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39372 deadline: 1680670666088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:46,097 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=175 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/95a16fa9862949898af4c62528b39c85 2023-04-05 04:56:46,101 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/cb3d50f46248404bac70b1229f52c270 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/cb3d50f46248404bac70b1229f52c270 2023-04-05 04:56:46,105 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/cb3d50f46248404bac70b1229f52c270, entries=150, sequenceid=175, filesize=25.3 K 2023-04-05 04:56:46,106 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/d2d9e490fdb24f1abd1f1e836db84078 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/d2d9e490fdb24f1abd1f1e836db84078 2023-04-05 04:56:46,110 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/d2d9e490fdb24f1abd1f1e836db84078, entries=150, sequenceid=175, filesize=11.8 K 2023-04-05 04:56:46,111 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/95a16fa9862949898af4c62528b39c85 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/95a16fa9862949898af4c62528b39c85 2023-04-05 04:56:46,114 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/95a16fa9862949898af4c62528b39c85, entries=150, sequenceid=175, filesize=11.8 K 2023-04-05 04:56:46,115 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 0a496ef74db8da540124a4d88d6e37aa in 462ms, sequenceid=175, compaction requested=true 2023-04-05 04:56:46,115 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:56:46,115 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:46,115 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:46,115 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:46,115 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:46,115 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:46,115 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:46,116 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:46,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:46,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:46,116 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:46,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:46,117 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:46,117 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:46,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:46,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:46,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:46,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:46,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:46,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:46,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a4fd0c5[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:46,118 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:46,118 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:46,119 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:46,119 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:46,119 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:46,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:46,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:46,119 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:46,119 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:46,119 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:46,120 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:46,120 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:46,120 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:46,120 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:46,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:46,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:46,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:46,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:46,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:46,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:46,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:46,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:46,123 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:46,123 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:46,123 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:46,123 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:46,123 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:46,124 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:46,124 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:46,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:46,124 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:46,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:46,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:46,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:46,125 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:46,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:46,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:46,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:46,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:46,126 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:46,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:46,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:46,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:46,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:46,127 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:46,127 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:46,127 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:46,127 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:46,127 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:46,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:46,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:46,127 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:46,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:46,127 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:46,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:46,128 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:46,128 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:46,128 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:46,128 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:46,128 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:46,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:46,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:46,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:46,228 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:46,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:46,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:46,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:46,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:46,231 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:46,231 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:46,231 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:46,231 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:46,231 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:46,231 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:46,232 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:46,232 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:46,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:46,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:46,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:46,232 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:46,232 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:46,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:46,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:46,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:46,233 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:46,233 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:46,233 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:46,233 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:46,233 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:46,233 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:46,233 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:46,233 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:46,233 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:46,234 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:46,234 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0a496ef74db8da540124a4d88d6e37aa 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-05 04:56:46,234 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=A 2023-04-05 04:56:46,234 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:46,234 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=B 2023-04-05 04:56:46,234 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:46,234 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=C 2023-04-05 04:56:46,234 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:46,247 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304058e8d43be1b6240bba53a0ccbde9625bf to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304058e8d43be1b6240bba53a0ccbde9625bf 2023-04-05 04:56:46,257 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=193, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/36464768e580455581164ad48366969d 2023-04-05 04:56:46,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:46,394 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:46,403 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:46,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39398 deadline: 1680670666399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:46,403 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:46,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39372 deadline: 1680670666399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:46,403 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:46,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39386 deadline: 1680670666400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:46,405 DEBUG [Thread-907] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4149 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa., hostname=jenkins-hbase20.apache.org,38091,1680670569206, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 04:56:46,405 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:46,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39406 deadline: 1680670666403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:46,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:46,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39422 deadline: 1680670666403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:46,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:46,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39372 deadline: 1680670666505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:46,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:46,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39386 deadline: 1680670666506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:46,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:46,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39406 deadline: 1680670666506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:46,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:46,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39422 deadline: 1680670666507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:46,669 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/f676873c28724ed9a538b9ab64ac871f 2023-04-05 04:56:46,709 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:46,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39372 deadline: 1680670666708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:46,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:46,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39406 deadline: 1680670666708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:46,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:46,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39422 deadline: 1680670666709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:46,712 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:46,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39386 deadline: 1680670666708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:47,013 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:47,013 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:47,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39372 deadline: 1680670667012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:47,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39406 deadline: 1680670667013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:47,015 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:47,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39422 deadline: 1680670667014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:47,016 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:47,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39386 deadline: 1680670667015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:47,083 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/c36e47081a37470e96c0c5d0c0588a45 2023-04-05 04:56:47,086 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/36464768e580455581164ad48366969d as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/36464768e580455581164ad48366969d 2023-04-05 04:56:47,090 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/36464768e580455581164ad48366969d, entries=150, sequenceid=193, filesize=25.3 K 2023-04-05 04:56:47,091 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/f676873c28724ed9a538b9ab64ac871f as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/f676873c28724ed9a538b9ab64ac871f 2023-04-05 04:56:47,095 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/f676873c28724ed9a538b9ab64ac871f, entries=150, sequenceid=193, filesize=11.8 K 2023-04-05 04:56:47,096 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/c36e47081a37470e96c0c5d0c0588a45 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/c36e47081a37470e96c0c5d0c0588a45 2023-04-05 04:56:47,104 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/c36e47081a37470e96c0c5d0c0588a45, entries=150, sequenceid=193, filesize=11.8 K 2023-04-05 04:56:47,105 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 0a496ef74db8da540124a4d88d6e37aa in 871ms, sequenceid=193, compaction requested=true 2023-04-05 04:56:47,105 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:56:47,105 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:47,105 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:47,105 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:47,105 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:47,110 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:47,110 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:47,111 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:47,112 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:47,112 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:47,112 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:47,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:47,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:47,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:47,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:47,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:47,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:47,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:47,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:47,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:47,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@579691ca[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:47,113 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:47,113 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:47,114 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:47,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:47,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:47,114 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:47,114 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:47,114 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:47,114 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:47,114 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:47,115 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:47,115 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:47,115 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:47,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:47,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:47,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:47,115 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:47,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:47,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:47,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:47,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:47,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:47,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:47,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:47,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:47,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:47,118 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:47,118 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:47,118 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:47,118 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:47,118 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:47,119 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:47,119 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:47,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:47,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:47,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:47,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:47,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:47,119 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:47,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:47,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:47,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:47,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:47,119 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:47,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:47,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:47,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:47,120 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:47,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:47,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:47,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:47,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:47,122 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:47,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:47,122 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:47,122 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:47,122 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:47,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:47,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:47,122 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:47,122 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:47,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:47,122 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:47,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:47,122 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:47,123 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:47,122 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:47,123 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:47,123 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:47,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:47,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:47,223 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:47,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:47,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:47,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:47,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:47,226 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:47,226 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:47,226 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:47,226 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:47,226 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:47,226 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:47,227 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:47,227 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:47,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:47,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:47,227 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:47,227 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:47,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:47,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:47,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:47,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:47,228 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:47,228 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:47,228 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:47,228 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:47,228 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:47,228 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:47,228 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:47,228 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:47,228 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:47,228 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:47,228 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0a496ef74db8da540124a4d88d6e37aa 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-05 04:56:47,229 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=A 2023-04-05 04:56:47,229 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:47,229 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=B 2023-04-05 04:56:47,229 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:47,229 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=C 2023-04-05 04:56:47,229 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:47,243 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304053322db6bb18b41689165133622788a9a to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304053322db6bb18b41689165133622788a9a 2023-04-05 04:56:47,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:47,517 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:47,532 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:47,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39386 deadline: 1680670667529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:47,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:47,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39406 deadline: 1680670667531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:47,536 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:47,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39372 deadline: 1680670667532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:47,536 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:47,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39422 deadline: 1680670667533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:47,634 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:47,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39386 deadline: 1680670667634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:47,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:47,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39406 deadline: 1680670667636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:47,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:47,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39372 deadline: 1680670667636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:47,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:47,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39422 deadline: 1680670667636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:47,654 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=207, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/8b53e8a2bb9144e68198c0ef561bc21e 2023-04-05 04:56:47,837 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:47,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39386 deadline: 1680670667836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:47,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:47,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39406 deadline: 1680670667841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:47,844 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:47,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39422 deadline: 1680670667841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:47,844 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:47,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39372 deadline: 1680670667841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,065 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=207 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/1e9c46f4bcf34d76af03314ee7766493 2023-04-05 04:56:48,075 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=207 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/f7d64b92a0aa4465ba21225ad9278315 2023-04-05 04:56:48,080 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/8b53e8a2bb9144e68198c0ef561bc21e as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/8b53e8a2bb9144e68198c0ef561bc21e 2023-04-05 04:56:48,090 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/8b53e8a2bb9144e68198c0ef561bc21e, entries=150, sequenceid=207, filesize=25.3 K 2023-04-05 04:56:48,092 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/1e9c46f4bcf34d76af03314ee7766493 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/1e9c46f4bcf34d76af03314ee7766493 2023-04-05 04:56:48,097 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/1e9c46f4bcf34d76af03314ee7766493, entries=150, sequenceid=207, filesize=11.8 K 2023-04-05 04:56:48,098 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/f7d64b92a0aa4465ba21225ad9278315 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/f7d64b92a0aa4465ba21225ad9278315 2023-04-05 04:56:48,102 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/f7d64b92a0aa4465ba21225ad9278315, entries=150, sequenceid=207, filesize=11.8 K 2023-04-05 04:56:48,103 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 0a496ef74db8da540124a4d88d6e37aa in 875ms, sequenceid=207, compaction requested=true 2023-04-05 04:56:48,103 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:56:48,103 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:48,103 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:48,103 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:48,103 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,107 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:48,107 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:48,108 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,108 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:48,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:48,108 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,108 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:48,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:48,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:48,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:48,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:48,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e7efdef[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:48,110 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:48,110 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,111 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,111 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:48,111 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:48,111 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:48,111 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,111 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:48,112 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:48,113 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:48,113 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:48,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:48,113 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:48,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:48,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:48,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:48,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:48,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:48,115 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:48,115 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:48,115 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:48,115 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:48,115 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:48,116 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,116 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:48,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:48,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,116 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:48,117 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:48,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:48,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:48,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:48,118 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:48,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,125 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,125 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:48,125 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:48,125 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:48,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:48,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:48,126 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:48,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:48,125 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:48,126 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:48,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:48,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:48,125 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,126 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:48,126 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,126 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,126 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:48,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:48,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:48,142 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0a496ef74db8da540124a4d88d6e37aa 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-05 04:56:48,142 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=A 2023-04-05 04:56:48,142 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:48,143 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=B 2023-04-05 04:56:48,143 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:48,143 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=C 2023-04-05 04:56:48,143 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:48,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:48,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39422 deadline: 1680670668155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:48,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39406 deadline: 1680670668155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:48,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39372 deadline: 1680670668156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:48,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39386 deadline: 1680670668156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,226 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:48,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:48,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:48,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:48,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:48,229 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:48,229 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:48,229 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:48,229 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:48,229 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,229 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:48,230 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:48,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:48,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:48,230 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:48,230 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,230 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:48,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:48,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:48,231 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:48,231 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:48,231 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:48,231 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:48,231 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:48,231 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:48,231 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:48,231 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:48,231 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:48,231 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:48,231 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:48,231 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:48,232 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:48,232 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:48,232 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,232 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:48,232 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:48,232 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,233 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:48,233 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,233 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:48,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:48,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49044f23[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:48,234 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:48,234 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,234 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,235 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:48,235 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:48,235 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:48,235 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:48,235 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:48,235 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:48,238 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,238 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,238 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:48,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:48,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:48,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:48,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:48,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:48,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:48,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:48,240 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:48,240 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:48,240 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:48,240 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:48,240 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:48,242 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,242 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:48,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:48,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,242 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:48,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:48,242 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:48,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:48,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:48,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,243 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:48,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,245 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:48,245 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:48,245 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:48,245 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:48,245 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:48,245 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:48,245 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,245 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,245 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,245 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:48,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:48,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:48,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:48,246 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:48,246 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:48,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:48,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:48,260 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:48,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39422 deadline: 1680670668259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,260 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:48,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39406 deadline: 1680670668259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,260 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:48,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39372 deadline: 1680670668259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,261 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:48,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39386 deadline: 1680670668259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,346 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:48,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:48,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:48,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:48,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:48,348 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:48,348 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:48,349 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:48,349 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:48,349 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,349 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:48,349 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:48,349 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:48,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:48,350 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,350 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:48,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:48,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:48,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:48,350 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:48,351 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:48,351 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:48,351 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:48,351 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:48,351 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:48,351 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:48,351 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:48,351 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:48,351 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:48,351 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:48,351 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:48,351 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:48,351 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:48,351 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,351 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:48,351 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:48,352 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,352 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:48,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:48,352 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,352 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:48,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:48,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:48,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:48,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:48,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d69673e[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:48,353 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:48,353 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,354 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,354 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:48,354 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:48,354 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:48,354 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,354 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:48,355 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:48,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:48,355 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:48,355 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:48,355 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:48,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:48,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:48,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:48,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:48,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:48,356 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:48,356 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:48,356 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:48,356 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:48,356 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:48,357 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,357 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,357 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:48,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:48,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:48,358 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:48,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:48,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:48,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:48,358 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:48,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,360 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:48,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:48,360 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:48,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:48,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:48,360 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:48,360 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:48,360 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:48,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:48,360 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:48,360 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:48,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:48,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:48,360 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,360 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,360 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,360 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,360 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:48,460 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:48,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:48,461 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:48,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39406 deadline: 1680670668461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:48,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:48,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:48,464 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:48,464 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:48,464 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:48,464 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:48,464 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,464 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:48,464 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:48,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39386 deadline: 1680670668461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,465 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:48,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39372 deadline: 1680670668462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,465 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:48,465 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,465 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:48,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39422 deadline: 1680670668462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:48,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:48,465 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,465 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:48,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:48,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:48,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:48,466 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:48,466 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:48,466 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:48,466 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:48,466 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:48,466 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:48,466 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:48,466 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:48,466 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:48,466 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:48,467 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:48,467 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:48,467 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:48,467 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:48,467 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,467 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:48,467 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:48,467 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,467 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:48,468 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,468 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:48,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:48,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e12b398[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:48,469 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:48,469 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,470 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,470 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:48,470 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:48,470 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:48,470 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,470 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:48,471 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,471 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:48,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:48,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:48,471 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:48,471 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:48,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:48,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:48,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:48,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:48,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:48,473 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:48,473 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:48,473 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:48,473 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:48,473 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:48,474 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,474 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,474 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:48,475 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:48,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:48,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:48,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:48,476 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:48,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,477 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:48,477 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:48,477 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:48,477 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:48,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:48,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:48,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:48,477 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:48,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:48,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:48,477 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:48,477 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:48,477 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:48,478 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:48,478 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,478 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,478 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,555 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304053732d9dc097846c08fa2e91aef136ee1 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304053732d9dc097846c08fa2e91aef136ee1 2023-04-05 04:56:48,578 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:48,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:48,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:48,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:48,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:48,580 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:48,580 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:48,581 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:48,581 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:48,581 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,581 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:48,582 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,582 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,582 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:48,582 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:48,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:48,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:48,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:48,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:48,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:48,583 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:48,583 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:48,583 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:48,583 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:48,583 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:48,583 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:48,584 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:48,584 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:48,584 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:48,584 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:48,584 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:48,584 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:48,584 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:48,584 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:48,584 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,584 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:48,584 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:48,585 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,585 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:48,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:48,585 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,585 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:48,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:48,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:48,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:48,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:48,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27aaa630[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:48,588 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:48,588 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,589 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,589 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:48,589 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:48,589 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,589 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:48,589 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:48,590 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:48,590 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:48,590 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,590 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:48,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:48,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:48,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:48,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:48,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:48,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:48,592 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:48,592 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:48,592 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:48,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:48,592 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:48,592 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:48,593 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,593 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:48,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:48,593 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:48,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:48,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:48,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:48,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,594 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:48,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,595 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:48,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,596 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,596 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:48,596 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:48,596 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:48,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:48,596 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:48,597 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:48,596 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:48,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:48,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:48,597 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,597 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,597 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,597 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:48,597 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:48,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:48,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:48,697 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:48,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:48,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:48,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:48,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:48,700 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:48,700 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:48,700 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:48,700 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:48,700 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,700 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:48,701 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:48,701 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:48,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:48,701 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,701 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:48,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:48,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:48,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:48,702 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:48,702 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:48,702 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:48,702 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:48,702 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:48,702 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:48,703 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:48,703 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:48,703 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:48,703 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:48,703 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:48,703 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:48,703 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:48,703 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:48,703 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,703 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:48,703 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:48,704 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,704 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:48,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:48,704 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,704 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:48,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:48,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:48,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:48,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:48,705 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:48,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c3c88b1[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:48,705 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,706 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,706 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:48,706 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:48,706 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:48,706 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,706 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:48,707 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:48,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:48,707 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:48,707 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:48,707 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:48,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:48,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:48,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:48,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:48,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:48,709 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:48,709 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:48,709 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:48,709 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:48,709 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:48,710 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,710 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:48,710 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:48,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:48,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:48,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:48,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:48,711 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:48,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,712 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:48,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,714 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:48,714 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:48,714 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,714 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:48,714 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:48,714 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,714 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,714 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,714 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:48,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:48,714 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:48,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:48,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:48,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:48,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:48,715 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:48,715 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:48,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:48,769 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:48,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39406 deadline: 1680670668764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,769 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:48,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39386 deadline: 1680670668765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,769 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:48,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39422 deadline: 1680670668766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,771 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:48,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39372 deadline: 1680670668768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,815 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:48,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:48,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:48,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:48,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:48,818 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:48,818 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:48,818 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:48,818 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:48,818 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,818 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:48,819 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:48,819 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:48,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:48,819 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,820 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:48,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:48,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:48,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:48,820 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:48,821 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:48,821 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:48,821 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:48,821 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:48,821 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:48,821 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:48,821 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:48,821 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:48,821 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:48,821 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:48,821 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:48,821 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:48,821 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:48,821 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,821 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:48,821 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:48,822 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,822 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:48,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:48,823 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,823 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:48,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:48,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:48,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:48,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:48,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a4a569f[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:48,824 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:48,824 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,824 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,824 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:48,824 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:48,824 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,824 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:48,824 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:48,825 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,825 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:48,825 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:48,825 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:48,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:48,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:48,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:48,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:48,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:48,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:48,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:48,827 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:48,827 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:48,827 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:48,827 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:48,827 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:48,828 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,828 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,828 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:48,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:48,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:48,829 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:48,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:48,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:48,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:48,830 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:48,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,832 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:48,832 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:48,832 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:48,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:48,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:48,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:48,832 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:48,832 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:48,832 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:48,832 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:48,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:48,832 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:48,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:48,832 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:48,832 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,832 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,832 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,833 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,932 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:48,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:48,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:48,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:48,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:48,936 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:48,936 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:48,936 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:48,936 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:48,937 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,937 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:48,937 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:48,937 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:48,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:48,937 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,938 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:48,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:48,938 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:48,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:48,938 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:48,938 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:48,938 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:48,938 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:48,938 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:48,938 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:48,939 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:48,939 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:48,939 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:48,939 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:48,939 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:48,939 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:48,939 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:48,939 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:48,939 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,939 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:48,939 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:48,940 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,940 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:48,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:48,940 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,940 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:48,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:48,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:48,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:48,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:48,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@715fb63d[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:48,942 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:48,942 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,943 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,943 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:48,943 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:48,943 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:48,943 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,943 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:48,943 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:48,943 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,943 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:48,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:48,944 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:48,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:48,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:48,945 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:48,946 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:48,946 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:48,946 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:48,946 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:48,946 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,946 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:48,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:48,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,947 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:48,947 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:48,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:48,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:48,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:48,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:48,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:48,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,949 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,950 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:48,950 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:48,950 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:48,950 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:48,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:48,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:48,950 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:48,950 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:48,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:48,950 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:48,950 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,950 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:48,950 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:48,951 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:48,951 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:48,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:48,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:48,961 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=227, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/ff90dd0a7b4e45e8921d3a0b1a20b2fa 2023-04-05 04:56:49,051 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:49,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:49,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:49,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:49,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:49,054 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:49,054 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:49,054 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:49,054 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:49,054 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,054 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:49,055 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:49,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:49,055 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:49,055 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,055 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:49,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:49,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:49,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:49,056 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:49,056 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:49,056 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:49,056 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:49,056 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:49,056 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:49,056 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:49,056 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:49,056 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:49,056 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:49,056 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:49,056 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:49,056 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:49,056 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:49,056 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,056 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:49,056 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:49,057 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,057 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:49,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:49,057 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:49,058 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:49,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:49,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:49,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:49,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b60500[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:49,058 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:49,059 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,059 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,059 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:49,059 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:49,059 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:49,059 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,059 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:49,065 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:49,065 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,065 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:49,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:49,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:49,065 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:49,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:49,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:49,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:49,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:49,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:49,068 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:49,068 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:49,068 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:49,068 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:49,068 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:49,069 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,069 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:49,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:49,069 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:49,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:49,069 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:49,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:49,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:49,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,075 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:49,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,084 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,084 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:49,084 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:49,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:49,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:49,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:49,084 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:49,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:49,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:49,084 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:49,084 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:49,084 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:49,084 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,084 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:49,084 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:49,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:49,084 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,084 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,184 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:49,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:49,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:49,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:49,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:49,188 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:49,188 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:49,188 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:49,188 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:49,188 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,188 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:49,189 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:49,189 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:49,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:49,189 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,189 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:49,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:49,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:49,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:49,190 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:49,190 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:49,190 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:49,190 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:49,190 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:49,190 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:49,190 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:49,190 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:49,190 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:49,190 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:49,190 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:49,190 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:49,191 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:49,191 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:49,191 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,191 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:49,191 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:49,191 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,191 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:49,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:49,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:49,192 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,192 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:49,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:49,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:49,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:49,193 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:49,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2be77cc7[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:49,193 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,193 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,193 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:49,193 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:49,193 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:49,193 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,194 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:49,194 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:49,194 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,194 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:49,194 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:49,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:49,196 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:49,196 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:49,196 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:49,196 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:49,196 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:49,197 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,197 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:49,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:49,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,197 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:49,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,197 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:49,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:49,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:49,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:49,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,198 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,200 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,200 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:49,200 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:49,200 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,200 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,200 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,200 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,200 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:49,200 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:49,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:49,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:49,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:49,200 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:49,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:49,201 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:49,201 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:49,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:49,275 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:49,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39406 deadline: 1680670669270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,275 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:49,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39386 deadline: 1680670669270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,276 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:49,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39372 deadline: 1680670669273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:49,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39422 deadline: 1680670669273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,301 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:49,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:49,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:49,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:49,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:49,304 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:49,304 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:49,304 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:49,304 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:49,304 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,304 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:49,305 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:49,305 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:49,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:49,305 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,305 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:49,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:49,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:49,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:49,306 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:49,306 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:49,306 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:49,306 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:49,306 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:49,306 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:49,306 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:49,306 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:49,306 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:49,306 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:49,306 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:49,306 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:49,306 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:49,306 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:49,306 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,306 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:49,306 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:49,307 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,307 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:49,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:49,307 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,307 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:49,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:49,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:49,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:49,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:49,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cee1360[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:49,309 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:49,309 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,310 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,310 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:49,310 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:49,310 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:49,311 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,311 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:49,311 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,311 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:49,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:49,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:49,311 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:49,312 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:49,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:49,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:49,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:49,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:49,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:49,313 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:49,314 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:49,314 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:49,314 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:49,314 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:49,315 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,315 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:49,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,315 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:49,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:49,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:49,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:49,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:49,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,316 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:49,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,317 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:49,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,319 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,319 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:49,319 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:49,319 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:49,319 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:49,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:49,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:49,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:49,319 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:49,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:49,319 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:49,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:49,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:49,319 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:49,319 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,320 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,319 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:49,320 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,378 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=227 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/62c08d8e072e44239c8de07c4e3f6e88 2023-04-05 04:56:49,420 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:49,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:49,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:49,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:49,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:49,423 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:49,423 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:49,423 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:49,423 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:49,423 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,423 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:49,424 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:49,424 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:49,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:49,424 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,424 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:49,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:49,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:49,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:49,425 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:49,425 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:49,425 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:49,425 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:49,425 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:49,425 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:49,425 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:49,425 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:49,425 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:49,425 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:49,425 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:49,425 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:49,425 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:49,425 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:49,425 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,425 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:49,425 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:49,426 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,426 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:49,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:49,426 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,426 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:49,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:49,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:49,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:49,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:49,427 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:49,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cff437e[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:49,427 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,428 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,428 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:49,428 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:49,428 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,428 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:49,428 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:49,429 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:49,429 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:49,429 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:49,429 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:49,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:49,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:49,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:49,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:49,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:49,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:49,431 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:49,431 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:49,431 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:49,431 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:49,431 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:49,432 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,432 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:49,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:49,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,432 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:49,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:49,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:49,432 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:49,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:49,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,433 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:49,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,434 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,435 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:49,435 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:49,435 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:49,435 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:49,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:49,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:49,435 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,435 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:49,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:49,435 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:49,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:49,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:49,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:49,435 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:49,435 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:49,435 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,436 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,536 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:49,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:49,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:49,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:49,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:49,539 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:49,539 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:49,539 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:49,539 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:49,539 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,539 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:49,540 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:49,540 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:49,540 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,540 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:49,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:49,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:49,541 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:49,541 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:49,541 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:49,541 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:49,541 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:49,541 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:49,541 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:49,541 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:49,541 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:49,541 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:49,542 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:49,542 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:49,542 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:49,542 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:49,542 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,542 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:49,542 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:49,543 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,543 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:49,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:49,543 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,543 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:49,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:49,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:49,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:49,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:49,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2688c012[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:49,544 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:49,544 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,545 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,545 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:49,545 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,545 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:49,545 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:49,545 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:49,546 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,546 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:49,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:49,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:49,546 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:49,546 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:49,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:49,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:49,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:49,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:49,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:49,547 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:49,547 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:49,548 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:49,548 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:49,548 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:49,548 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,548 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:49,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:49,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,548 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:49,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,549 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:49,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:49,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:49,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:49,550 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:49,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,551 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,551 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:49,551 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:49,551 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:49,551 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:49,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:49,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:49,551 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,551 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,551 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,551 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:49,551 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:49,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:49,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:49,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:49,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:49,552 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:49,552 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:49,652 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:49,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:49,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:49,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:49,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:49,655 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:49,655 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:49,655 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:49,655 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:49,655 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,655 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:49,656 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:49,656 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:49,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:49,656 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,656 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:49,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:49,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:49,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:49,657 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:49,657 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:49,657 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:49,657 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:49,657 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:49,657 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:49,657 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:49,657 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:49,657 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:49,657 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:49,657 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:49,657 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:49,657 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:49,657 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:49,657 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,657 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:49,658 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:49,659 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,659 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:49,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:49,659 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,659 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:49,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:49,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:49,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:49,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:49,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2514383d[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:49,660 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:49,661 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,661 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,661 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:49,661 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:49,661 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:49,661 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,661 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:49,662 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,662 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:49,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:49,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:49,662 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:49,662 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:49,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:49,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:49,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:49,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:49,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:49,664 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:49,664 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:49,664 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:49,664 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:49,664 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:49,665 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,665 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:49,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:49,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,665 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:49,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:49,665 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:49,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:49,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:49,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,666 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:49,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,668 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:49,668 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:49,668 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:49,668 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:49,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:49,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:49,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:49,668 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:49,668 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:49,668 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:49,668 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:49,668 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:49,668 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:49,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:49,668 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,668 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,768 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:49,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:49,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:49,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:49,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:49,771 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:49,771 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:49,771 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:49,771 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:49,771 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,771 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:49,772 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,772 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:49,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:49,772 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,772 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:49,772 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:49,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:49,773 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:49,773 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:49,773 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:49,773 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:49,773 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:49,773 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:49,773 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:49,773 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:49,773 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:49,773 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:49,773 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:49,773 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:49,773 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:49,773 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:49,773 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,773 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:49,773 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:49,774 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,774 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:49,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:49,774 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,774 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:49,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:49,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:49,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:49,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:49,776 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:49,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79bd5aec[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:49,776 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,776 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,776 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:49,776 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:49,776 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:49,776 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,776 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:49,777 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:49,777 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,777 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:49,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:49,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:49,777 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:49,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:49,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:49,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:49,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:49,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:49,779 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:49,779 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:49,779 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:49,779 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:49,779 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:49,780 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,780 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,780 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:49,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:49,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:49,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:49,780 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:49,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:49,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:49,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,781 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:49,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:49,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,783 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,783 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:49,783 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:49,783 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:49,783 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:49,783 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:49,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:49,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:49,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:49,783 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:49,783 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:49,783 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:49,783 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,783 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:49,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:49,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:49,783 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:49,789 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=227 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/77531efe7c1a40658c837c4fc3e458ea 2023-04-05 04:56:49,793 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/ff90dd0a7b4e45e8921d3a0b1a20b2fa as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/ff90dd0a7b4e45e8921d3a0b1a20b2fa 2023-04-05 04:56:49,797 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/ff90dd0a7b4e45e8921d3a0b1a20b2fa, entries=200, sequenceid=227, filesize=32.1 K 2023-04-05 04:56:49,798 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/62c08d8e072e44239c8de07c4e3f6e88 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/62c08d8e072e44239c8de07c4e3f6e88 2023-04-05 04:56:49,802 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/62c08d8e072e44239c8de07c4e3f6e88, entries=150, sequenceid=227, filesize=11.8 K 2023-04-05 04:56:49,803 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/77531efe7c1a40658c837c4fc3e458ea as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/77531efe7c1a40658c837c4fc3e458ea 2023-04-05 04:56:49,807 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/77531efe7c1a40658c837c4fc3e458ea, entries=150, sequenceid=227, filesize=11.8 K 2023-04-05 04:56:49,808 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 0a496ef74db8da540124a4d88d6e37aa in 1665ms, sequenceid=227, compaction requested=true 2023-04-05 04:56:49,808 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:56:49,808 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:56:49,808 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-05 04:56:49,808 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-05 04:56:49,808 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:56:49,808 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:56:49,810 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 195240 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-05 04:56:49,810 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 84240 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-05 04:56:49,810 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): 0a496ef74db8da540124a4d88d6e37aa/A is initiating minor compaction (all files) 2023-04-05 04:56:49,810 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0a496ef74db8da540124a4d88d6e37aa/A in TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:49,810 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/46255102075e4d8a82f802be2ab91ddf, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/6565982749d040e69836b3fc37c0ad05, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/2604088d799a4626a5262d47fcca7a83, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/cb3d50f46248404bac70b1229f52c270, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/36464768e580455581164ad48366969d, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/8b53e8a2bb9144e68198c0ef561bc21e, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/ff90dd0a7b4e45e8921d3a0b1a20b2fa] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp, totalSize=190.7 K 2023-04-05 04:56:49,811 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): 0a496ef74db8da540124a4d88d6e37aa/B is initiating minor compaction (all files) 2023-04-05 04:56:49,811 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0a496ef74db8da540124a4d88d6e37aa/B in TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:49,811 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/9b0cf2a048f14847ba39afe51d58b0d7, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/f191857faca74ddc8baae6cf23c40e89, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/78a13d97b8d840bca3fbc69c706c30e6, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/d2d9e490fdb24f1abd1f1e836db84078, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/f676873c28724ed9a538b9ab64ac871f, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/1e9c46f4bcf34d76af03314ee7766493, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/62c08d8e072e44239c8de07c4e3f6e88] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp, totalSize=82.3 K 2023-04-05 04:56:49,811 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 46255102075e4d8a82f802be2ab91ddf, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1680670602233 2023-04-05 04:56:49,811 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 9b0cf2a048f14847ba39afe51d58b0d7, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1680670602233 2023-04-05 04:56:49,811 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 6565982749d040e69836b3fc37c0ad05, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1680670602245 2023-04-05 04:56:49,811 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting f191857faca74ddc8baae6cf23c40e89, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1680670602245 2023-04-05 04:56:49,811 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 2604088d799a4626a5262d47fcca7a83, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1680670603258 2023-04-05 04:56:49,812 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 78a13d97b8d840bca3fbc69c706c30e6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1680670603258 2023-04-05 04:56:49,812 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting cb3d50f46248404bac70b1229f52c270, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=175, earliestPutTs=1680670605417 2023-04-05 04:56:49,812 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting d2d9e490fdb24f1abd1f1e836db84078, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=175, earliestPutTs=1680670605417 2023-04-05 04:56:49,812 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 36464768e580455581164ad48366969d, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1680670605769 2023-04-05 04:56:49,812 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting f676873c28724ed9a538b9ab64ac871f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1680670605769 2023-04-05 04:56:49,812 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 8b53e8a2bb9144e68198c0ef561bc21e, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1680670606394 2023-04-05 04:56:49,813 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 1e9c46f4bcf34d76af03314ee7766493, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1680670606394 2023-04-05 04:56:49,813 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting ff90dd0a7b4e45e8921d3a0b1a20b2fa, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1680670607526 2023-04-05 04:56:49,813 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 62c08d8e072e44239c8de07c4e3f6e88, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1680670607526 2023-04-05 04:56:49,828 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): 0a496ef74db8da540124a4d88d6e37aa#B#compaction#147 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:56:49,838 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0a496ef74db8da540124a4d88d6e37aa#A#compaction#146 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:56:49,851 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/a051711b31ce44c98c704efc4c1e4cda as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/a051711b31ce44c98c704efc4c1e4cda 2023-04-05 04:56:49,855 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 0a496ef74db8da540124a4d88d6e37aa/B of 0a496ef74db8da540124a4d88d6e37aa into a051711b31ce44c98c704efc4c1e4cda(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:56:49,855 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:56:49,855 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa., storeName=0a496ef74db8da540124a4d88d6e37aa/B, priority=9, startTime=1680670609808; duration=0sec 2023-04-05 04:56:49,855 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:56:49,855 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-05 04:56:49,862 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 84240 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-05 04:56:49,862 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): 0a496ef74db8da540124a4d88d6e37aa/C is initiating minor compaction (all files) 2023-04-05 04:56:49,862 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0a496ef74db8da540124a4d88d6e37aa/C in TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:49,862 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/077e1e0832f84261b1b78b6990e361ba, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/19c08e36c73444e58a2684a11148ed88, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/89cc91b1a22a4b11895f83b5b00a5eb7, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/95a16fa9862949898af4c62528b39c85, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/c36e47081a37470e96c0c5d0c0588a45, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/f7d64b92a0aa4465ba21225ad9278315, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/77531efe7c1a40658c837c4fc3e458ea] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp, totalSize=82.3 K 2023-04-05 04:56:49,864 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 077e1e0832f84261b1b78b6990e361ba, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1680670602233 2023-04-05 04:56:49,864 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 19c08e36c73444e58a2684a11148ed88, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1680670602245 2023-04-05 04:56:49,864 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 89cc91b1a22a4b11895f83b5b00a5eb7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1680670603258 2023-04-05 04:56:49,865 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 95a16fa9862949898af4c62528b39c85, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=175, earliestPutTs=1680670605417 2023-04-05 04:56:49,865 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting c36e47081a37470e96c0c5d0c0588a45, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1680670605769 2023-04-05 04:56:49,866 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting f7d64b92a0aa4465ba21225ad9278315, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1680670606394 2023-04-05 04:56:49,866 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 77531efe7c1a40658c837c4fc3e458ea, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1680670607526 2023-04-05 04:56:49,881 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): 0a496ef74db8da540124a4d88d6e37aa#C#compaction#148 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:56:49,883 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:49,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:49,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:49,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:49,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:49,887 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:49,887 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:49,887 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:49,887 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:49,887 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,887 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:49,888 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,888 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:49,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:49,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:49,888 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:49,888 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:49,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:49,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:49,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:49,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:49,889 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:49,894 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:49,894 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:49,894 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:49,894 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:49,895 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:49,895 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:49,895 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:49,895 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:49,895 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:49,895 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0a496ef74db8da540124a4d88d6e37aa 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-05 04:56:49,895 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=A 2023-04-05 04:56:49,895 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:49,895 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=B 2023-04-05 04:56:49,895 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:49,895 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=C 2023-04-05 04:56:49,896 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:50,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:50,280 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:50,298 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/dcc201d119cd4e74b64f050a90e45f98 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/dcc201d119cd4e74b64f050a90e45f98 2023-04-05 04:56:50,300 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/da76bf31651e4f37bcaf08d34f531a6f as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/da76bf31651e4f37bcaf08d34f531a6f 2023-04-05 04:56:50,303 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:50,303 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:50,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39406 deadline: 1680670670299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:50,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39386 deadline: 1680670670300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:50,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670610296, max timestamp=1680670610302Num uniques 859; 2023-04-05 04:56:50,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:56:50,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670610296, max timestamp=1680670610302Num uniques 859; 2023-04-05 04:56:50,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670610296, max timestamp=1680670610302Num uniques 859; 2023-04-05 04:56:50,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:56:50,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:56:50,305 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 0a496ef74db8da540124a4d88d6e37aa/C of 0a496ef74db8da540124a4d88d6e37aa into dcc201d119cd4e74b64f050a90e45f98(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:56:50,305 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:56:50,305 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa., storeName=0a496ef74db8da540124a4d88d6e37aa/C, priority=9, startTime=1680670609808; duration=0sec 2023-04-05 04:56:50,305 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:56:50,307 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 0a496ef74db8da540124a4d88d6e37aa/A of 0a496ef74db8da540124a4d88d6e37aa into da76bf31651e4f37bcaf08d34f531a6f(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:56:50,307 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:56:50,307 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa., storeName=0a496ef74db8da540124a4d88d6e37aa/A, priority=9, startTime=1680670609808; duration=0sec 2023-04-05 04:56:50,307 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:56:50,315 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405631a013f702d459cb03dd5e0c78200af to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405631a013f702d459cb03dd5e0c78200af 2023-04-05 04:56:50,333 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:50,333 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:50,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39372 deadline: 1680670670329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:50,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39422 deadline: 1680670670330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:50,408 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:50,408 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:50,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39386 deadline: 1680670670405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:50,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39406 deadline: 1680670670405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:50,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:50,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39398 deadline: 1680670670410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:50,412 DEBUG [Thread-907] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8157 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa., hostname=jenkins-hbase20.apache.org,38091,1680670569206, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 04:56:50,436 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:50,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39422 deadline: 1680670670434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:50,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:50,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39372 deadline: 1680670670434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:50,612 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:50,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39406 deadline: 1680670670609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:50,612 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:50,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39386 deadline: 1680670670609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:50,639 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:50,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39372 deadline: 1680670670638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:50,640 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:50,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39422 deadline: 1680670670639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:50,729 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=240, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/1d7d840151e447fa8fe72710e4eba529 2023-04-05 04:56:50,744 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/fbeeb3e026794f89a4cbdf7a2eef7a66 2023-04-05 04:56:50,915 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:50,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39386 deadline: 1680670670914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:50,915 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:50,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39406 deadline: 1680670670915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:50,942 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:50,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39372 deadline: 1680670670940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:50,943 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:50,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39422 deadline: 1680670670942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,154 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/ba47dea7ed2b4fa2960d21e610211d10 2023-04-05 04:56:51,159 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/1d7d840151e447fa8fe72710e4eba529 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/1d7d840151e447fa8fe72710e4eba529 2023-04-05 04:56:51,165 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/1d7d840151e447fa8fe72710e4eba529, entries=150, sequenceid=240, filesize=25.3 K 2023-04-05 04:56:51,167 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/fbeeb3e026794f89a4cbdf7a2eef7a66 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/fbeeb3e026794f89a4cbdf7a2eef7a66 2023-04-05 04:56:51,172 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/fbeeb3e026794f89a4cbdf7a2eef7a66, entries=150, sequenceid=240, filesize=11.8 K 2023-04-05 04:56:51,173 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/ba47dea7ed2b4fa2960d21e610211d10 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/ba47dea7ed2b4fa2960d21e610211d10 2023-04-05 04:56:51,178 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/ba47dea7ed2b4fa2960d21e610211d10, entries=150, sequenceid=240, filesize=11.8 K 2023-04-05 04:56:51,179 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 0a496ef74db8da540124a4d88d6e37aa in 1284ms, sequenceid=240, compaction requested=false 2023-04-05 04:56:51,179 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:56:51,179 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:51,180 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:51,180 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:51,180 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,180 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:51,180 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:51,181 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,181 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,181 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:51,183 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:51,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:51,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:51,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:51,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:51,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:51,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:51,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f1f1216[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:51,187 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:51,187 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,192 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,192 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:51,192 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:51,192 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:51,192 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,193 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:51,193 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,193 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:51,193 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:51,193 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:51,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:51,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:51,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:51,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:51,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:51,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:51,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:51,196 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:51,196 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:51,196 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:51,196 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:51,196 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:51,197 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,197 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:51,197 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,198 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:51,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:51,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:51,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:51,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:51,199 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:51,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:51,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:51,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,202 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:51,202 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:51,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:51,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:51,202 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,202 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:51,202 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:51,202 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,203 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:51,203 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:51,203 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,203 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:51,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:51,203 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:51,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:51,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:51,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:51,203 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:51,303 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:51,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:51,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:51,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:51,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:51,306 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:51,306 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:51,306 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:51,306 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:51,307 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,307 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:51,307 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:51,307 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:51,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:51,308 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,308 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:51,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:51,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:51,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:51,308 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:51,309 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:51,309 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:51,309 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:51,309 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:51,309 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:51,309 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:51,309 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:51,309 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:51,309 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:51,309 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0a496ef74db8da540124a4d88d6e37aa 3/3 column families, dataSize=120.76 KB heapSize=340.45 KB 2023-04-05 04:56:51,310 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=A 2023-04-05 04:56:51,310 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:56:51,310 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=B 2023-04-05 04:56:51,310 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:56:51,310 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=C 2023-04-05 04:56:51,310 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:56:51,346 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405d4aa31a9784a4295af42b3fb27dedf45 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405d4aa31a9784a4295af42b3fb27dedf45 2023-04-05 04:56:51,369 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=276, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/931e200b679b482ab38b691433368987 2023-04-05 04:56:51,396 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=276 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/0bddf914895941afb72a53e6eb170c09 2023-04-05 04:56:51,423 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=276 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/c61fec32fc0c4ab9aaf7623e50a3151e 2023-04-05 04:56:51,424 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:51,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:51,432 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/931e200b679b482ab38b691433368987 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/931e200b679b482ab38b691433368987 2023-04-05 04:56:51,439 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/931e200b679b482ab38b691433368987, entries=150, sequenceid=276, filesize=25.4 K 2023-04-05 04:56:51,440 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/0bddf914895941afb72a53e6eb170c09 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/0bddf914895941afb72a53e6eb170c09 2023-04-05 04:56:51,444 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/0bddf914895941afb72a53e6eb170c09, entries=150, sequenceid=276, filesize=11.9 K 2023-04-05 04:56:51,445 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:51,445 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/c61fec32fc0c4ab9aaf7623e50a3151e as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/c61fec32fc0c4ab9aaf7623e50a3151e 2023-04-05 04:56:51,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39406 deadline: 1680670671440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,446 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:51,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39372 deadline: 1680670671444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:51,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39386 deadline: 1680670671445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:51,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39422 deadline: 1680670671446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,451 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/c61fec32fc0c4ab9aaf7623e50a3151e, entries=150, sequenceid=276, filesize=11.9 K 2023-04-05 04:56:51,452 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.41 KB/348576, currentSize=60.38 KB/61830 for 0a496ef74db8da540124a4d88d6e37aa in 143ms, sequenceid=276, compaction requested=true 2023-04-05 04:56:51,452 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:56:51,452 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:51,452 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:51,452 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:51,453 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,453 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:51,453 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:51,453 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,453 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:51,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:51,454 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,454 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:51,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:51,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:51,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:51,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:51,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:51,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ff7ddb7[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:51,455 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:51,455 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,457 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,457 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:51,457 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,457 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:51,457 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:51,457 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:51,458 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:51,458 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:51,458 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,458 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:51,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:51,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:51,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:51,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:51,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:51,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:51,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:51,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:51,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:51,460 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:51,460 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:51,460 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:51,460 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:51,460 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:51,462 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,462 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:51,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:51,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,463 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:51,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:51,463 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:51,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:51,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:51,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:51,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:51,464 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:51,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:51,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,465 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:51,465 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:51,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:51,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:51,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:51,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:51,466 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,466 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:51,466 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:51,466 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,466 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,466 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,466 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,466 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:51,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:51,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:51,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:51,466 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:51,466 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:51,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:51,548 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0a496ef74db8da540124a4d88d6e37aa 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-05 04:56:51,548 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=A 2023-04-05 04:56:51,549 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:51,549 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=B 2023-04-05 04:56:51,549 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:51,549 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=C 2023-04-05 04:56:51,549 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:51,567 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:51,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:51,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:51,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:51,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:51,569 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:51,569 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:51,569 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:51,569 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:51,569 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,570 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:51,570 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:51,570 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:51,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:51,570 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,571 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:51,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:51,571 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:51,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:51,571 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:51,571 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:51,571 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:51,571 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:51,571 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:51,572 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:51,572 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:51,572 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:51,572 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:51,572 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:51,572 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:51,572 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:51,572 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:51,572 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,572 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:51,572 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:51,573 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,573 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:51,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:51,573 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,573 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:51,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:51,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:51,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:51,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:51,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:51,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ba6c3a4[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:51,574 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:51,574 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,575 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,575 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:51,575 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:51,575 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:51,575 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,575 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:51,576 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,576 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:51,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:51,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:51,576 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:51,576 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:51,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:51,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:51,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:51,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:51,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:51,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:51,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:51,578 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:51,578 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:51,578 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:51,578 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:51,578 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:51,579 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,579 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:51,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:51,579 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:51,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:51,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:51,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:51,580 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,580 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:51,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:51,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39406 deadline: 1680670671578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,580 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:51,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39386 deadline: 1680670671578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:51,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:51,581 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:51,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:51,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,582 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,582 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:51,582 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:51,582 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:51,582 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:51,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:51,582 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:51,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:51,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:51,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:51,582 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:51,582 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,583 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,583 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:51,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:51,583 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:51,583 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:51,682 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:51,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39406 deadline: 1680670671681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,682 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:51,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39386 deadline: 1680670671681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,683 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:51,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:51,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:51,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:51,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:51,686 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:51,686 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:51,686 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:51,686 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:51,686 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,686 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:51,687 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,687 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:51,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:51,687 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,687 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:51,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:51,687 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:51,688 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:51,688 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:51,688 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:51,688 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:51,688 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:51,688 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:51,688 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:51,688 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:51,688 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:51,688 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:51,688 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:51,688 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:51,688 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:51,688 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,688 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:51,688 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:51,689 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,689 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:51,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:51,689 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,689 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:51,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:51,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:51,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:51,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:51,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:51,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48df9806[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:51,690 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:51,690 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,691 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,691 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:51,691 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:51,691 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:51,691 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,691 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:51,691 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,691 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:51,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:51,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:51,692 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:51,692 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:51,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:51,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:51,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:51,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:51,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:51,693 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:51,693 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:51,693 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:51,693 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:51,693 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:51,694 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:51,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:51,694 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,694 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:51,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:51,694 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:51,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:51,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:51,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:51,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:51,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,695 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:51,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:51,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,697 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,697 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:51,697 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:51,697 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,697 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,697 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:51,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:51,697 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:51,697 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:51,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:51,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:51,697 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,698 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:51,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:51,698 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:51,698 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:51,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:51,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:51,798 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:51,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:51,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:51,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:51,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:51,802 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:51,802 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:51,802 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:51,802 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:51,802 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,802 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:51,803 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,803 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:51,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:51,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:51,803 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,803 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:51,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:51,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:51,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:51,804 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:51,804 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:51,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:51,804 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:51,804 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:51,804 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:51,804 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:51,804 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:51,804 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:51,804 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:51,805 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:51,805 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:51,805 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:51,805 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:51,805 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,805 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:51,805 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:51,805 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,805 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:51,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:51,806 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,806 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:51,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:51,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:51,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:51,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:51,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:51,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f17bca9[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:51,807 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:51,807 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,807 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,807 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:51,807 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:51,807 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,807 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:51,807 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:51,809 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:51,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:51,809 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:51,809 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:51,809 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:51,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:51,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:51,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:51,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:51,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:51,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:51,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:51,811 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:51,811 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:51,811 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:51,811 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:51,811 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:51,811 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,812 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:51,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,812 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:51,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:51,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:51,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:51,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:51,812 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:51,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:51,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:51,814 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:51,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:51,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,815 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:51,815 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:51,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:51,815 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:51,815 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:51,816 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:51,816 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,816 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:51,816 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:51,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:51,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:51,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:51,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:51,816 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,816 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,816 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:51,816 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:51,884 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:51,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39386 deadline: 1680670671883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,884 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:51,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39406 deadline: 1680670671883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,916 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:51,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:51,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:51,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:51,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:51,919 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:51,919 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:51,919 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:51,919 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:51,919 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,919 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:51,920 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,920 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:51,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:51,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:51,920 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,920 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:51,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:51,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:51,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:51,921 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:51,921 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:51,921 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:51,921 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:51,921 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:51,921 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:51,921 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:51,921 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:51,921 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:51,922 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:51,922 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:51,922 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:51,922 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:51,922 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:51,922 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,922 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:51,922 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:51,923 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,923 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:51,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:51,923 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,923 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:51,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:51,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:51,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:51,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:51,924 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:51,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e6cc299[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:51,924 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,924 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,925 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:51,925 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,925 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:51,925 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:51,925 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:51,925 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:51,925 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:51,925 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,925 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:51,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:51,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:51,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:51,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:51,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:51,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:51,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:51,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:51,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:51,927 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:51,927 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:51,927 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:51,927 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:51,927 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:51,928 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,928 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,928 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:51,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:51,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:51,929 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:51,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:51,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:51,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:51,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:51,929 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:51,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:51,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:51,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,931 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,931 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:51,931 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:51,931 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:51,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:51,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:51,931 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:51,931 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:51,931 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:51,931 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:51,931 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,931 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:51,931 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:51,931 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:51,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:51,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:51,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:51,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:51,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:51,961 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405cd461254d87a41b3a67b014b80fa1118 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405cd461254d87a41b3a67b014b80fa1118 2023-04-05 04:56:52,031 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:52,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:52,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:52,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:52,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:52,034 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:52,034 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:52,034 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:52,034 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:52,034 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,034 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,035 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:52,035 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:52,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:52,035 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,035 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:52,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,036 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:52,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,036 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:52,036 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:52,036 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:52,036 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:52,036 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:52,036 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:52,037 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:52,037 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:52,037 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:52,037 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:52,037 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:52,037 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:52,037 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:52,037 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:52,037 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,037 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:52,037 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:52,038 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,038 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:52,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:52,038 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,038 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:52,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:52,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:52,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:52,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:52,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ba9713[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:52,039 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:52,039 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,040 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,040 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:52,040 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:52,040 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:52,040 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,040 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:52,041 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,041 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:52,041 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:52,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:52,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:52,041 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:52,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:52,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:52,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:52,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:52,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:52,043 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:52,043 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:52,043 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:52,043 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:52,043 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:52,044 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:52,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:52,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:52,044 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,044 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:52,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,044 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:52,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:52,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:52,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,045 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:52,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,046 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,046 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,046 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,046 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,046 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,047 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,047 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,046 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:52,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:52,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:52,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:52,047 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:52,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:52,047 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:52,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:52,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:52,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:52,049 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:52,049 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:52,149 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:52,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:52,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:52,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:52,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:52,152 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:52,152 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:52,152 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:52,152 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:52,152 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,152 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,153 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:52,153 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:52,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:52,153 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,153 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:52,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:52,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,154 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:52,154 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:52,154 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:52,154 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:52,154 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:52,154 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:52,154 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:52,154 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:52,154 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:52,155 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:52,155 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:52,155 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:52,155 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:52,155 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:52,155 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,155 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:52,155 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:52,156 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,156 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:52,156 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,156 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:52,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:52,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@371120ca[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:52,157 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:52,157 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,158 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,158 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:52,158 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:52,158 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:52,158 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,158 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:52,159 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,159 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:52,159 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:52,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:52,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:52,159 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:52,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:52,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:52,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:52,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:52,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:52,161 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:52,161 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:52,161 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:52,161 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:52,161 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:52,162 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,162 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:52,162 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:52,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:52,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:52,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:52,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:52,162 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:52,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,164 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:52,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,166 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:52,166 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:52,166 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:52,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:52,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:52,166 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,167 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,167 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,167 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,167 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,167 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,167 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:52,167 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:52,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:52,167 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:52,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:52,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:52,188 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:52,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39406 deadline: 1680670672186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,188 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:52,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39386 deadline: 1680670672186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,267 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:52,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:52,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:52,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:52,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:52,270 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:52,270 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:52,270 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:52,270 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:52,270 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,270 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,271 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:52,271 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:52,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:52,271 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,271 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:52,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:52,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,272 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:52,272 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:52,272 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:52,272 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:52,272 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:52,272 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:52,272 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:52,272 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:52,272 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:52,272 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:52,272 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:52,272 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:52,273 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:52,273 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:52,273 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,273 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:52,273 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:52,273 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,273 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:52,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:52,274 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,274 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:52,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:52,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:52,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:52,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:52,275 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:52,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5513075[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:52,275 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,275 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,275 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:52,275 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:52,276 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:52,276 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,276 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:52,276 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,276 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:52,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:52,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:52,276 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:52,276 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:52,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:52,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:52,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:52,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:52,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:52,278 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:52,278 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:52,278 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:52,278 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:52,278 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:52,279 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,279 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:52,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:52,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,279 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:52,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:52,279 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:52,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:52,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:52,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,280 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:52,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,282 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,282 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,282 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:52,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:52,282 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:52,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:52,282 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,282 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,282 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,282 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,282 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:52,282 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:52,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:52,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:52,282 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:52,282 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:52,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:52,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:52,366 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=289, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/9f31d589737e4cb5882709f205a51ac4 2023-04-05 04:56:52,383 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:52,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:52,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:52,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:52,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:52,385 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:52,385 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:52,385 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:52,385 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:52,386 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,386 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,386 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,386 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:52,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:52,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:52,386 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,386 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:52,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:52,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,387 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:52,387 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:52,387 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:52,387 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:52,387 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:52,387 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:52,387 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:52,387 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:52,387 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:52,387 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:52,388 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:52,388 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:52,388 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:52,388 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:52,388 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,388 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:52,388 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:52,388 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,388 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:52,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:52,389 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,389 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:52,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:52,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d21424c[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:52,389 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:52,390 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,390 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,390 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:52,390 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:52,390 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,390 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:52,390 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:52,391 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,391 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:52,391 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:52,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:52,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:52,391 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:52,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:52,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:52,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:52,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:52,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:52,393 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:52,393 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:52,393 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:52,393 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:52,393 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:52,394 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,394 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,394 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:52,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:52,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:52,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:52,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:52,394 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:52,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:52,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,395 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:52,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,397 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,397 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,397 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:52,397 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:52,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:52,397 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:52,397 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,397 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,397 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,397 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:52,397 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:52,397 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:52,397 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:52,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:52,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:52,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:52,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:52,451 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:52,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39372 deadline: 1680670672451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,459 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:52,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39422 deadline: 1680670672457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,497 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:52,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:52,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:52,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:52,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:52,500 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:52,500 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:52,500 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:52,500 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:52,500 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,500 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,501 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,501 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:52,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:52,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:52,501 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,501 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:52,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,502 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:52,502 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:52,502 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:52,502 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:52,502 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:52,502 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:52,502 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:52,502 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:52,502 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:52,502 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:52,503 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:52,503 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:52,503 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:52,503 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:52,503 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,503 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:52,503 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:52,503 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,503 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:52,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:52,504 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,504 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:52,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:52,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:52,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:52,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:52,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5af44ef4[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:52,505 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:52,506 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,506 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,507 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:52,507 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:52,507 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:52,507 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,507 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:52,507 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:52,507 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,507 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:52,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:52,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:52,507 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:52,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:52,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:52,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:52,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:52,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:52,509 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:52,509 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:52,509 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:52,509 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:52,509 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:52,510 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,510 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:52,510 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:52,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:52,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:52,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:52,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:52,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,511 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:52,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,512 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:52,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,514 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,514 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,514 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,514 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,514 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:52,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:52,514 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:52,514 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:52,514 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,514 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,514 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:52,514 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:52,514 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:52,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:52,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:52,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:52,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:52,614 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:52,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:52,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:52,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:52,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:52,617 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:52,617 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:52,617 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:52,617 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:52,617 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,617 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,618 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,618 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:52,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:52,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:52,618 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,618 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:52,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:52,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:52,619 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:52,619 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:52,619 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:52,619 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:52,620 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:52,620 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:52,620 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:52,620 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:52,620 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:52,620 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:52,620 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:52,620 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:52,620 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:52,620 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,620 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:52,620 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:52,621 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:52,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:52,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:52,621 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:52,622 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,622 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:52,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:52,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:52,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64dd3210[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:52,622 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:52,622 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,623 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,623 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,623 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:52,623 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:52,624 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:52,624 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:52,624 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:52,624 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:52,624 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,624 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:52,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:52,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:52,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:52,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:52,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:52,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:52,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:52,626 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:52,626 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:52,626 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:52,626 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:52,626 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:52,627 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,627 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:52,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:52,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,627 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:52,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:52,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:52,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:52,628 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:52,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,629 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:52,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,630 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,630 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:52,630 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,630 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,630 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:52,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:52,630 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:52,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:52,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:52,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:52,630 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,630 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:52,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:52,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:52,630 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:52,630 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,630 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:52,630 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,690 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:52,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39406 deadline: 1680670672689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,690 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:52,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39386 deadline: 1680670672689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,731 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:52,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:52,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:52,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:52,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:52,733 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:52,733 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:52,733 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:52,734 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:52,734 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,734 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,734 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,734 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:52,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:52,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:52,735 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,735 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:52,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:52,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,735 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:52,735 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:52,735 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:52,736 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:52,736 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:52,736 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:52,736 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:52,736 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:52,736 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:52,736 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:52,736 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:52,736 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:52,736 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:52,736 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:52,736 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,736 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:52,736 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:52,737 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,737 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:52,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:52,737 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,737 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:52,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:52,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:52,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:52,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:52,738 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:52,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68836466[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:52,738 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,739 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,739 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,739 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:52,739 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:52,739 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:52,739 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:52,739 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:52,739 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:52,739 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,739 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:52,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:52,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:52,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:52,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:52,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:52,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:52,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:52,741 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:52,741 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:52,741 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:52,741 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:52,741 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:52,742 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,742 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:52,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,742 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:52,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:52,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:52,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:52,742 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:52,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:52,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,743 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:52,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,745 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,745 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,745 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:52,745 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:52,745 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:52,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:52,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:52,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:52,745 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:52,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:52,745 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,745 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,745 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,745 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:52,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:52,745 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:52,745 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:52,775 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=289 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/e8f783f3e4db494b8bc89d16fc98b9d5 2023-04-05 04:56:52,845 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:52,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:52,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:52,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:52,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:52,848 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:52,848 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:52,848 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:52,848 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:52,849 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,849 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,849 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,849 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:52,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:52,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:52,850 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,850 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:52,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:52,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,850 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:52,851 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:52,851 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:52,851 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:52,851 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:52,851 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:52,851 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:52,851 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:52,851 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:52,851 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:52,851 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:52,851 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:52,851 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:52,851 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:52,851 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,851 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:52,851 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:52,852 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,852 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:52,852 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,852 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:52,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:52,853 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:52,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fb77818[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:52,853 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,854 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,854 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:52,854 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:52,854 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,854 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:52,854 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:52,855 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:52,855 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:52,855 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,855 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:52,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:52,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:52,857 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:52,857 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:52,857 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:52,857 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:52,857 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:52,857 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,857 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:52,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,857 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:52,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:52,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:52,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:52,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:52,858 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:52,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,859 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:52,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,860 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,861 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,861 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,861 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,861 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:52,861 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:52,861 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:52,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:52,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:52,861 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,861 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,861 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:52,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:52,861 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:52,861 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:52,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:52,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:52,961 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:52,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:52,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:52,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:52,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:52,964 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:52,964 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:52,964 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:52,964 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:52,964 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,964 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,966 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:52,966 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:52,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:52,966 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,966 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:52,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:52,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,966 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:52,967 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:52,967 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:52,967 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:52,967 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:52,967 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:52,967 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:52,967 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:52,967 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:52,967 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:52,967 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:52,967 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:52,967 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:52,967 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:52,967 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,967 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:52,967 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:52,968 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,968 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:52,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:52,968 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:52,968 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:52,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:52,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:52,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:52,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@174c18f4[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:52,969 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:52,969 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,970 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,970 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:52,970 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:52,970 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,970 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:52,970 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:52,971 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,971 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:52,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:52,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:52,971 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:52,971 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:52,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:52,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:52,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:52,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:52,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:52,972 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:52,973 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:52,973 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:52,973 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:52,973 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:52,973 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,973 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:52,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:52,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,973 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:52,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,974 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:52,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:52,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:52,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:52,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,974 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:52,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:52,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,976 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,976 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:52,976 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:52,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:52,976 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:52,976 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:52,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:52,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:52,976 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:52,976 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:52,976 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,976 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:52,976 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:52,976 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:52,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:52,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:52,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:52,976 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:53,077 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:53,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:53,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:53,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:53,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:53,079 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:53,079 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:53,079 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:53,079 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:53,080 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:53,080 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:53,080 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:53,080 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:53,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:53,080 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:53,080 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:53,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:53,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:53,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:53,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:53,081 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:53,081 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:53,081 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:53,081 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:53,081 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:53,081 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:53,081 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:53,082 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:53,082 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:53,082 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:53,082 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:53,082 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:53,082 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:53,082 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:53,082 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:53,082 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:53,082 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:53,083 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:53,083 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:53,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:53,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:53,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:53,083 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:53,083 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:53,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:53,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:53,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:53,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:53,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:53,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:53,084 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:53,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a0ce5bf[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:53,084 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:53,085 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:53,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:53,085 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:53,085 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:53,085 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:53,085 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:53,085 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:53,086 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:53,086 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:53,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:53,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:53,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:53,086 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:53,086 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:53,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:53,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:53,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:53,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:53,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:53,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:53,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:53,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:53,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:53,088 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:53,088 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:53,088 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:53,088 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:53,088 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:53,089 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:53,089 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:53,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:53,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:53,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:53,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:53,089 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:53,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:53,090 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:53,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:53,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:53,091 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:53,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:53,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:53,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:53,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:53,092 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:53,092 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:53,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:53,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:53,092 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:53,092 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:53,092 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:53,092 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:53,092 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:53,092 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:53,092 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:53,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:53,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:53,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:53,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:53,092 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:53,093 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:53,093 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:53,093 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:53,185 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=289 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/1e2a79dd37d04e13b5625d918f7965db 2023-04-05 04:56:53,189 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/9f31d589737e4cb5882709f205a51ac4 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/9f31d589737e4cb5882709f205a51ac4 2023-04-05 04:56:53,193 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:53,193 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/9f31d589737e4cb5882709f205a51ac4, entries=200, sequenceid=289, filesize=32.3 K 2023-04-05 04:56:53,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:53,194 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/e8f783f3e4db494b8bc89d16fc98b9d5 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/e8f783f3e4db494b8bc89d16fc98b9d5 2023-04-05 04:56:53,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:53,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:53,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:53,195 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:53,196 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:53,196 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:53,196 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:53,196 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:53,196 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:53,196 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:53,196 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:53,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:53,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:53,197 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:53,197 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:53,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:53,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:53,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:53,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:53,197 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:53,197 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:53,198 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:53,198 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:53,198 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:53,198 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:53,198 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:53,198 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:53,198 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:53,198 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:53,198 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:53,198 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:53,198 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:53,198 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:53,198 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:53,198 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:53,199 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:53,199 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:53,199 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:53,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:53,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:53,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:53,200 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:53,200 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:53,200 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/e8f783f3e4db494b8bc89d16fc98b9d5, entries=150, sequenceid=289, filesize=11.9 K 2023-04-05 04:56:53,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:53,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:53,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:53,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:53,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:53,201 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/1e2a79dd37d04e13b5625d918f7965db as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/1e2a79dd37d04e13b5625d918f7965db 2023-04-05 04:56:53,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:53,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69df12cc[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:53,201 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:53,201 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:53,201 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:53,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:53,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:53,202 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:53,202 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:53,202 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:53,202 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:53,202 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:53,202 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:53,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:53,203 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:53,203 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:53,203 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:53,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:53,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:53,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:53,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:53,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:53,205 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/1e2a79dd37d04e13b5625d918f7965db, entries=150, sequenceid=289, filesize=11.9 K 2023-04-05 04:56:53,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:53,205 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-05 04:56:53,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:53,205 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:53,205 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:53,205 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:53,205 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:53,205 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:53,206 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 0a496ef74db8da540124a4d88d6e37aa in 1658ms, sequenceid=289, compaction requested=true 2023-04-05 04:56:53,206 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:56:53,206 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:56:53,206 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:56:53,206 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:56:53,206 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:56:53,206 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:56:53,206 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:53,206 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:53,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:53,206 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:53,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:53,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:53,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:53,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:53,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:53,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:53,207 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:53,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:53,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:53,207 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111300 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:56:53,208 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): 0a496ef74db8da540124a4d88d6e37aa/A is initiating minor compaction (all files) 2023-04-05 04:56:53,208 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0a496ef74db8da540124a4d88d6e37aa/A in TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:53,208 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/da76bf31651e4f37bcaf08d34f531a6f, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/1d7d840151e447fa8fe72710e4eba529, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/931e200b679b482ab38b691433368987, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/9f31d589737e4cb5882709f205a51ac4] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp, totalSize=108.7 K 2023-04-05 04:56:53,208 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48820 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:56:53,208 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): 0a496ef74db8da540124a4d88d6e37aa/B is initiating minor compaction (all files) 2023-04-05 04:56:53,208 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0a496ef74db8da540124a4d88d6e37aa/B in TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:53,208 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/a051711b31ce44c98c704efc4c1e4cda, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/fbeeb3e026794f89a4cbdf7a2eef7a66, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/0bddf914895941afb72a53e6eb170c09, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/e8f783f3e4db494b8bc89d16fc98b9d5] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp, totalSize=47.7 K 2023-04-05 04:56:53,208 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting da76bf31651e4f37bcaf08d34f531a6f, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1680670607526 2023-04-05 04:56:53,209 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting a051711b31ce44c98c704efc4c1e4cda, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1680670607526 2023-04-05 04:56:53,209 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 1d7d840151e447fa8fe72710e4eba529, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1680670608154 2023-04-05 04:56:53,209 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting fbeeb3e026794f89a4cbdf7a2eef7a66, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1680670608154 2023-04-05 04:56:53,209 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 931e200b679b482ab38b691433368987, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1680670610324 2023-04-05 04:56:53,209 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 0bddf914895941afb72a53e6eb170c09, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1680670610324 2023-04-05 04:56:53,210 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 9f31d589737e4cb5882709f205a51ac4, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1680670611436 2023-04-05 04:56:53,210 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting e8f783f3e4db494b8bc89d16fc98b9d5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1680670611436 2023-04-05 04:56:53,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:53,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:53,211 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:53,216 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:53,216 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:53,216 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:53,217 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:53,217 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:53,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:53,217 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:53,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:53,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:53,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:53,217 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:53,217 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:53,217 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:53,217 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:53,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:53,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:53,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:53,217 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:53,218 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:53,223 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): 0a496ef74db8da540124a4d88d6e37aa#B#compaction#158 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:56:53,231 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0a496ef74db8da540124a4d88d6e37aa#A#compaction#159 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:56:53,318 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:53,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:53,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:53,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:53,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:53,320 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:53,320 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:53,320 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:53,320 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:53,321 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:53,321 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:53,321 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:53,321 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:53,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:53,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:53,322 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:53,322 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:53,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:53,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:53,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:53,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:53,322 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:53,323 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:53,323 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:53,323 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:53,323 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:53,323 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:53,323 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:53,323 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:53,323 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:53,323 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:53,323 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0a496ef74db8da540124a4d88d6e37aa 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-05 04:56:53,323 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=A 2023-04-05 04:56:53,323 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:53,324 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=B 2023-04-05 04:56:53,324 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:53,324 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=C 2023-04-05 04:56:53,324 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:53,633 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/6cde669885f94d678cc09ba2e7e894d1 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/6cde669885f94d678cc09ba2e7e894d1 2023-04-05 04:56:53,638 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0a496ef74db8da540124a4d88d6e37aa/B of 0a496ef74db8da540124a4d88d6e37aa into 6cde669885f94d678cc09ba2e7e894d1(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:56:53,638 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:56:53,639 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa., storeName=0a496ef74db8da540124a4d88d6e37aa/B, priority=12, startTime=1680670613206; duration=0sec 2023-04-05 04:56:53,639 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:56:53,639 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:56:53,640 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48820 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:56:53,640 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): 0a496ef74db8da540124a4d88d6e37aa/C is initiating minor compaction (all files) 2023-04-05 04:56:53,640 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0a496ef74db8da540124a4d88d6e37aa/C in TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:53,641 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/dcc201d119cd4e74b64f050a90e45f98, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/ba47dea7ed2b4fa2960d21e610211d10, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/c61fec32fc0c4ab9aaf7623e50a3151e, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/1e2a79dd37d04e13b5625d918f7965db] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp, totalSize=47.7 K 2023-04-05 04:56:53,641 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting dcc201d119cd4e74b64f050a90e45f98, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1680670607526 2023-04-05 04:56:53,641 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting ba47dea7ed2b4fa2960d21e610211d10, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1680670608154 2023-04-05 04:56:53,641 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting c61fec32fc0c4ab9aaf7623e50a3151e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1680670610324 2023-04-05 04:56:53,642 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 1e2a79dd37d04e13b5625d918f7965db, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1680670611436 2023-04-05 04:56:53,651 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): 0a496ef74db8da540124a4d88d6e37aa#C#compaction#161 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:56:53,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:53,700 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:53,714 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:53,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39406 deadline: 1680670673712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:53,714 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:53,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39386 deadline: 1680670673713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:53,739 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304056a2d20dacdf44ca88e4d223087c40054 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304056a2d20dacdf44ca88e4d223087c40054 2023-04-05 04:56:53,816 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:53,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39386 deadline: 1680670673815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:53,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:53,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39406 deadline: 1680670673815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:54,019 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:54,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39386 deadline: 1680670674017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:54,019 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:54,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39406 deadline: 1680670674019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:54,061 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/ecce563d265d4fe5be6cfff1aedbf142 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/ecce563d265d4fe5be6cfff1aedbf142 2023-04-05 04:56:54,066 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0a496ef74db8da540124a4d88d6e37aa/C of 0a496ef74db8da540124a4d88d6e37aa into ecce563d265d4fe5be6cfff1aedbf142(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:56:54,066 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:56:54,066 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa., storeName=0a496ef74db8da540124a4d88d6e37aa/C, priority=12, startTime=1680670613206; duration=0sec 2023-04-05 04:56:54,066 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:56:54,144 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=309, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/f0a81c4988e0447c8cc04bc9878208f5 2023-04-05 04:56:54,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:54,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39386 deadline: 1680670674321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:54,323 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:54,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39406 deadline: 1680670674322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:54,453 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/7591d7f360dd487e85d6df1a74f9e833 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/7591d7f360dd487e85d6df1a74f9e833 2023-04-05 04:56:54,457 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0a496ef74db8da540124a4d88d6e37aa/A of 0a496ef74db8da540124a4d88d6e37aa into 7591d7f360dd487e85d6df1a74f9e833(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-05 04:56:54,457 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:56:54,457 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa., storeName=0a496ef74db8da540124a4d88d6e37aa/A, priority=12, startTime=1680670613206; duration=1sec 2023-04-05 04:56:54,458 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:56:54,464 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:54,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39372 deadline: 1680670674463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:54,465 DEBUG [Thread-911] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4135 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa., hostname=jenkins-hbase20.apache.org,38091,1680670569206, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 04:56:54,479 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:54,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39422 deadline: 1680670674478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:54,481 DEBUG [Thread-913] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4150 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa., hostname=jenkins-hbase20.apache.org,38091,1680670569206, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 04:56:54,554 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=309 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/416bec6712c043ca93221ecc364b4bd7 2023-04-05 04:56:54,825 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:54,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39406 deadline: 1680670674824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:54,827 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:54,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39386 deadline: 1680670674827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:54,965 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=309 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/1326761fd322418b964419f51e7e2b2b 2023-04-05 04:56:54,969 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/f0a81c4988e0447c8cc04bc9878208f5 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/f0a81c4988e0447c8cc04bc9878208f5 2023-04-05 04:56:54,972 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/f0a81c4988e0447c8cc04bc9878208f5, entries=150, sequenceid=309, filesize=25.4 K 2023-04-05 04:56:54,974 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/416bec6712c043ca93221ecc364b4bd7 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/416bec6712c043ca93221ecc364b4bd7 2023-04-05 04:56:54,978 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/416bec6712c043ca93221ecc364b4bd7, entries=150, sequenceid=309, filesize=11.9 K 2023-04-05 04:56:54,979 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/1326761fd322418b964419f51e7e2b2b as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/1326761fd322418b964419f51e7e2b2b 2023-04-05 04:56:54,983 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/1326761fd322418b964419f51e7e2b2b, entries=150, sequenceid=309, filesize=11.9 K 2023-04-05 04:56:54,986 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 0a496ef74db8da540124a4d88d6e37aa in 1663ms, sequenceid=309, compaction requested=false 2023-04-05 04:56:54,986 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:56:54,986 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:54,986 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:54,986 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:54,986 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:54,986 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:54,986 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:54,988 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:54,988 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:54,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:54,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:54,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:54,989 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:54,989 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:54,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:54,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:54,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:54,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:54,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:54,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:54,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d7cd057[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:54,990 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:54,990 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:54,991 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:54,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:54,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:54,991 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:54,991 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:54,991 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:54,991 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:54,991 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:54,992 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:54,992 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:54,992 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:54,992 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:54,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:54,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:54,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:54,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:54,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:54,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:54,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:54,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:54,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:54,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:54,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:54,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:54,995 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:54,995 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:54,995 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:54,995 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:54,995 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:54,996 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:54,996 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:54,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:54,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:54,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:54,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:54,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:54,996 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:54,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:54,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:54,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:54,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:54,997 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:54,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:54,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:54,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:54,999 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:54,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:54,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:55,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:55,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,001 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:55,001 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:55,001 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,001 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:55,002 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:55,002 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,002 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:55,002 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:55,002 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:55,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:55,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:55,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:55,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:55,002 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:55,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:55,002 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:55,002 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:55,102 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:55,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:55,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:55,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:55,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:55,105 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:55,105 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:55,105 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:55,105 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:55,105 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:55,105 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:55,106 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:55,106 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:55,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:55,106 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,106 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:55,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:55,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:55,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:55,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:55,107 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:55,107 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:55,107 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:55,107 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:55,107 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:55,107 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:55,107 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:55,107 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:55,108 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:55,108 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:55,108 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0a496ef74db8da540124a4d88d6e37aa 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-05 04:56:55,108 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=A 2023-04-05 04:56:55,108 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:55,108 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=B 2023-04-05 04:56:55,108 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:55,108 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=C 2023-04-05 04:56:55,108 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:55,132 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304056d4875fa382d48839b90e8162af9ea35 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304056d4875fa382d48839b90e8162af9ea35 2023-04-05 04:56:55,137 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=324, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/c35a623a4df741f6a739c56823c3af80 2023-04-05 04:56:55,152 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=324 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/b23a9dc69e804907acd292544fdb3675 2023-04-05 04:56:55,573 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=324 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/8e078cf80e414dbd85f3c0b918e2ba71 2023-04-05 04:56:55,577 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/c35a623a4df741f6a739c56823c3af80 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/c35a623a4df741f6a739c56823c3af80 2023-04-05 04:56:55,581 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/c35a623a4df741f6a739c56823c3af80, entries=150, sequenceid=324, filesize=25.4 K 2023-04-05 04:56:55,582 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/b23a9dc69e804907acd292544fdb3675 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/b23a9dc69e804907acd292544fdb3675 2023-04-05 04:56:55,585 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/b23a9dc69e804907acd292544fdb3675, entries=150, sequenceid=324, filesize=11.9 K 2023-04-05 04:56:55,586 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/8e078cf80e414dbd85f3c0b918e2ba71 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/8e078cf80e414dbd85f3c0b918e2ba71 2023-04-05 04:56:55,590 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/8e078cf80e414dbd85f3c0b918e2ba71, entries=150, sequenceid=324, filesize=11.9 K 2023-04-05 04:56:55,591 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 0a496ef74db8da540124a4d88d6e37aa in 483ms, sequenceid=324, compaction requested=true 2023-04-05 04:56:55,591 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:56:55,591 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:55,591 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:55,591 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:55,591 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,591 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:55,591 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:55,592 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,592 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:55,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:55,593 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:55,593 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:55,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:55,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:55,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a80f4b4[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:55,598 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:55,598 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:55,600 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:55,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:55,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:55,601 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:55,601 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:55,601 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,601 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:55,601 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:55,601 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:55,601 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:55,602 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:55,605 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:55,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:55,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:55,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:55,607 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:55,607 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:55,607 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:55,607 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:55,607 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:55,608 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:55,608 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:55,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:55,608 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:55,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:55,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:55,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:55,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:55,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:55,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:55,609 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:55,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:55,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:55,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:55,611 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:55,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:55,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:55,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,613 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,613 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:55,613 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:55,613 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:55,613 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:55,613 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:55,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:55,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:55,613 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,613 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:55,613 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:55,613 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:55,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:55,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:55,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:55,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:55,613 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:55,613 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:55,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:55,714 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:55,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:55,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:55,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:55,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:55,730 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:55,730 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:55,730 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:55,730 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:55,730 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:55,730 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:55,731 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:55,731 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:55,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:55,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:55,732 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,732 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:55,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:55,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:55,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:55,732 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:55,733 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:55,733 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:55,733 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:55,733 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:55,733 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:55,734 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:55,734 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:55,734 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:55,734 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:55,734 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:56:55,734 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:55,734 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:55,734 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:55,734 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,734 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:55,735 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:55,735 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:55,736 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:55,736 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:55,736 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:55,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:55,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:55,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:55,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37dcd146[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:55,738 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:55,738 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:55,738 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:55,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:55,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:55,738 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:55,738 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:55,738 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:55,739 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,739 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:55,739 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:55,739 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:55,739 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,739 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:55,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:55,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:55,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:55,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:55,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:55,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:55,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:55,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:55,743 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:55,743 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:55,743 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:55,743 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:55,743 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:55,750 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:55,750 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:55,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:55,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:55,750 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:55,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:55,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:55,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:55,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:55,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:55,750 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:55,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:55,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:55,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:55,751 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:55,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:55,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:55,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,759 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,759 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:55,759 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:55,759 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,759 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:55,759 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:55,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:55,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:55,759 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:55,759 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:55,759 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:55,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:55,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:55,759 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:55,759 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:55,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:55,759 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:55,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:55,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:55,860 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:55,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:55,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:55,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:55,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:55,863 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:55,863 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:55,863 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:55,863 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:55,864 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:55,864 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:55,864 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:55,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:55,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:55,864 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,865 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,865 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:55,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:55,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:55,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:55,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:55,865 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:55,866 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:55,866 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:55,866 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:55,866 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:55,866 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:55,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:55,875 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0a496ef74db8da540124a4d88d6e37aa 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 04:56:55,876 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=A 2023-04-05 04:56:55,877 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:55,877 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=B 2023-04-05 04:56:55,877 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:55,877 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=C 2023-04-05 04:56:55,877 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:55,882 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:55,882 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:55,882 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:55,882 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:55,882 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:55,882 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:55,882 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:55,882 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:55,882 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,882 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:55,882 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:55,884 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:55,884 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:55,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:55,884 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:55,884 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:55,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:55,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:55,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:55,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:55,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:55,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2604fc03[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:55,886 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:55,886 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:55,887 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:55,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:55,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:55,887 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:55,887 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:55,887 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:55,887 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,887 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:55,889 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:55,889 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:55,889 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:55,889 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:55,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:55,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:55,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:55,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:55,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:55,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:55,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:55,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:55,894 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:55,894 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:55,894 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:55,894 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:55,894 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:55,900 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:55,900 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:55,900 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:55,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:55,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:55,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:55,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:55,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:55,901 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:55,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:55,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:55,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:55,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:55,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:55,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:55,902 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:55,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:55,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,905 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:55,905 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:55,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:55,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:55,905 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:55,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:55,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:55,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:55,905 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:55,905 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:55,905 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:55,905 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:55,905 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:55,905 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,905 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:55,905 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:55,905 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:55,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670615905, max timestamp=1680670615916Num uniques 859; 2023-04-05 04:56:55,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:56:55,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670615905, max timestamp=1680670615916Num uniques 859; 2023-04-05 04:56:55,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:56:55,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670615905, max timestamp=1680670615916Num uniques 859; 2023-04-05 04:56:55,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:56:55,920 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:55,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39406 deadline: 1680670675914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:55,971 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:55,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39386 deadline: 1680670675967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,005 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:56,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:56,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:56,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:56,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:56,008 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:56,008 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:56,008 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:56,008 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:56,008 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,009 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,009 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,009 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:56,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:56,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:56,009 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,010 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:56,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:56,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,010 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:56,010 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:56,010 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:56,010 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:56,010 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:56,010 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:56,011 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:56,011 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:56,011 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:56,011 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:56,011 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:56,011 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:56,011 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:56,011 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:56,011 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,011 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:56,011 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:56,012 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,012 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:56,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:56,012 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,012 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:56,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:56,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:56,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:56,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:56,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7da16efc[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:56,013 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:56,013 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,014 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,014 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:56,014 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:56,014 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:56,014 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,014 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:56,015 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,015 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:56,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:56,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:56,015 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:56,015 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:56,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:56,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:56,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:56,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:56,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:56,016 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:56,016 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:56,016 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:56,016 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:56,016 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:56,017 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,017 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:56,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:56,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,017 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:56,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:56,018 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:56,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:56,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:56,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,018 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:56,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,020 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,020 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:56,020 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,020 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:56,020 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,020 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,020 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:56,020 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:56,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:56,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:56,020 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,020 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,020 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:56,020 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:56,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:56,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:56,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:56,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:56,023 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:56,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39406 deadline: 1680670676021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,074 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:56,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39386 deadline: 1680670676072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,120 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:56,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:56,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:56,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:56,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:56,123 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:56,123 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:56,123 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:56,123 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:56,124 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,124 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,124 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,124 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:56,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:56,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:56,124 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,124 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:56,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:56,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,125 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:56,125 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:56,125 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:56,125 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:56,125 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:56,126 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:56,126 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:56,126 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:56,126 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:56,126 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:56,126 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:56,126 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:56,126 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:56,126 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:56,126 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,126 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:56,126 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:56,127 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:56,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:56,127 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:56,127 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,127 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:56,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:56,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:56,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:56,128 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:56,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@304910a8[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:56,128 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,128 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,129 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:56,129 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:56,129 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:56,129 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,129 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:56,129 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,129 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:56,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:56,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:56,129 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:56,130 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:56,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:56,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:56,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:56,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:56,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:56,131 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:56,131 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:56,131 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:56,131 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:56,131 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:56,132 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,132 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,132 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:56,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:56,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:56,132 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:56,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:56,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:56,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:56,133 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:56,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,135 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:56,135 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:56,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:56,135 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:56,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:56,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:56,135 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:56,135 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:56,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:56,135 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,136 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,136 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:56,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:56,136 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:56,136 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,136 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,136 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,225 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:56,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39406 deadline: 1680670676225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,236 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:56,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:56,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:56,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:56,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:56,238 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:56,238 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:56,238 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:56,239 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:56,239 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,239 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,240 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:56,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:56,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:56,240 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,240 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,240 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:56,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:56,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,240 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:56,241 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:56,241 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:56,241 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:56,241 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:56,241 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:56,241 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:56,241 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:56,241 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:56,241 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:56,241 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:56,241 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:56,241 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:56,241 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:56,241 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,241 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:56,241 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:56,242 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,242 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:56,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:56,242 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,242 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:56,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:56,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:56,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:56,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:56,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@614eb106[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:56,243 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:56,243 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,244 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,244 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:56,244 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:56,244 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:56,244 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,244 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:56,245 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,245 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:56,245 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:56,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:56,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:56,245 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:56,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:56,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:56,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:56,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:56,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:56,247 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:56,247 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:56,247 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:56,247 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:56,247 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:56,249 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,249 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,249 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:56,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:56,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:56,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,249 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:56,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:56,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:56,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:56,250 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:56,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,251 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,251 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,251 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,251 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:56,251 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:56,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:56,251 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,251 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:56,252 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,252 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:56,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:56,252 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:56,252 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:56,252 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:56,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:56,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:56,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:56,278 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:56,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39386 deadline: 1680670676275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,313 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040542c712f9ef0c4f549e6e04fd06993e40 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040542c712f9ef0c4f549e6e04fd06993e40 2023-04-05 04:56:56,352 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:56,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:56,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:56,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:56,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:56,355 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:56,355 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:56,355 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:56,355 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:56,356 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,356 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,356 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:56,356 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:56,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:56,357 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,357 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:56,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:56,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,357 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:56,358 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:56,358 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:56,358 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:56,358 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:56,358 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:56,358 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:56,358 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:56,358 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:56,358 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:56,358 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:56,358 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:56,358 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:56,358 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:56,358 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,358 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:56,358 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:56,359 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,359 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:56,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:56,359 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,359 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:56,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:56,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:56,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:56,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:56,360 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:56,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60abadc8[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:56,360 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,361 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,361 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:56,361 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:56,361 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:56,361 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,361 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:56,361 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,361 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:56,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:56,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:56,361 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:56,362 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:56,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:56,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:56,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:56,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:56,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:56,363 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:56,363 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:56,363 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:56,363 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:56,363 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:56,364 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,364 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:56,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:56,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,364 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:56,364 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:56,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:56,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:56,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:56,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,365 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:56,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,367 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,367 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,367 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,367 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,367 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,367 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,367 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:56,367 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:56,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:56,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:56,367 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:56,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:56,367 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:56,367 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:56,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:56,367 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:56,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:56,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:56,467 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:56,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:56,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:56,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:56,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:56,470 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:56,470 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:56,470 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:56,470 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:56,470 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,470 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,471 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,471 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:56,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:56,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:56,471 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,471 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:56,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:56,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,472 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:56,472 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:56,472 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:56,472 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:56,472 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:56,472 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:56,472 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:56,472 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:56,472 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:56,472 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:56,472 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:56,472 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:56,472 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:56,472 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:56,472 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,472 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:56,472 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:56,473 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:56,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:56,473 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:56,473 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,473 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:56,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:56,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:56,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:56,474 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:56,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@160be9af[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:56,474 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,475 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,475 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:56,475 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:56,475 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:56,475 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,475 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:56,475 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,475 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:56,475 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:56,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:56,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:56,476 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:56,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:56,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:56,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:56,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:56,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:56,477 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:56,477 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:56,477 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:56,477 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:56,477 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:56,478 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,478 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:56,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,478 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:56,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:56,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:56,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:56,478 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:56,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:56,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,479 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:56,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,481 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:56,481 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,481 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:56,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:56,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:56,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:56,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:56,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:56,481 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:56,481 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,481 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,481 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,481 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,481 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:56,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:56,482 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:56,481 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:56,482 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,532 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:56,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39406 deadline: 1680670676529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,581 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:56,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39386 deadline: 1680670676581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,582 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:56,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:56,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:56,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:56,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:56,584 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:56,584 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:56,584 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:56,584 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:56,584 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,584 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,585 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,585 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:56,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:56,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:56,585 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,585 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:56,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:56,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,586 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:56,586 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:56,586 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:56,586 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:56,586 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:56,586 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:56,586 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:56,586 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:56,586 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:56,586 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:56,586 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:56,586 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:56,586 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:56,586 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:56,586 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,587 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:56,587 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:56,587 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,587 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:56,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:56,588 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,588 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:56,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:56,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:56,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:56,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:56,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e3f6796[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:56,589 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:56,589 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,590 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,590 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:56,590 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:56,590 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:56,590 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,590 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:56,590 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:56,591 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:56,591 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,591 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:56,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:56,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:56,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:56,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:56,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:56,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:56,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:56,592 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:56,592 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:56,592 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:56,592 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:56,592 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:56,593 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,593 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:56,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:56,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,593 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:56,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:56,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:56,593 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:56,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:56,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,594 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:56,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,596 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,596 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,596 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,596 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,596 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:56,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:56,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:56,596 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:56,596 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,596 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:56,597 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:56,597 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:56,597 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:56,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:56,597 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:56,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:56,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:56,697 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:56,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:56,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:56,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:56,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:56,699 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:56,700 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:56,700 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:56,700 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:56,700 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,700 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,701 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,701 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:56,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:56,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:56,701 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,701 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:56,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:56,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,702 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:56,702 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:56,702 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:56,702 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:56,702 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:56,702 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:56,702 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:56,702 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:56,702 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:56,702 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:56,702 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:56,702 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:56,702 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:56,702 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:56,702 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,702 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:56,702 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:56,703 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,703 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:56,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:56,703 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:56,704 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:56,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:56,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:56,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:56,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a6d2dd6[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:56,704 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:56,705 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,705 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,705 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:56,705 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:56,705 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,705 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:56,705 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:56,706 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:56,706 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:56,706 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,706 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:56,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:56,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:56,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:56,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:56,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:56,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:56,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:56,708 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:56,708 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:56,708 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:56,708 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:56,708 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:56,708 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:56,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:56,709 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,709 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:56,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:56,709 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:56,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:56,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:56,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,710 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:56,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,712 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,712 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:56,712 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,712 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:56,712 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:56,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:56,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:56,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:56,712 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:56,712 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:56,712 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,712 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:56,712 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,712 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:56,712 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:56,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:56,718 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=336, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/694443b181f34006a2a60f59779a616c 2023-04-05 04:56:56,812 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:56,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:56,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:56,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:56,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:56,815 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:56,815 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:56,815 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:56,815 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:56,816 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,816 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,816 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,816 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:56,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:56,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:56,816 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,817 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:56,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:56,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:56,817 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:56,817 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:56,817 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:56,817 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:56,817 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:56,818 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:56,818 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:56,818 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:56,818 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:56,818 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:56,818 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:56,818 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:56,818 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:56,818 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,818 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:56,818 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:56,819 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,819 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:56,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:56,819 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,819 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:56,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:56,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:56,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:56,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:56,820 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:56,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47151abe[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:56,820 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,821 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,821 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:56,821 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,821 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:56,821 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:56,821 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:56,822 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:56,822 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:56,822 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:56,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:56,822 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:56,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:56,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:56,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:56,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:56,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:56,824 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:56,824 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:56,824 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:56,824 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:56,824 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:56,825 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,825 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,825 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:56,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:56,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:56,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:56,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:56,825 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:56,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:56,826 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:56,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,827 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:56,828 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:56,828 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:56,828 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:56,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:56,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:56,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:56,828 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,828 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,828 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,828 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,828 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,828 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,828 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:56,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:56,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:56,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:56,828 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:56,928 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:56,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:56,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:56,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:56,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:56,931 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:56,931 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:56,931 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:56,931 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:56,931 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,931 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,932 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:56,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:56,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:56,932 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,932 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,933 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:56,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:56,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,933 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:56,933 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:56,933 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:56,933 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:56,933 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:56,933 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:56,933 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:56,933 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:56,933 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:56,933 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:56,934 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:56,934 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:56,934 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:56,934 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:56,934 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,934 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:56,934 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:56,935 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,935 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:56,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:56,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:56,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:56,935 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,935 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:56,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:56,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:56,936 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:56,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e9ac562[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:56,936 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,936 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,937 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:56,937 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:56,937 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:56,937 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,937 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:56,937 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,937 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:56,937 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:56,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:56,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:56,937 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:56,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:56,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:56,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:56,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:56,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:56,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:56,939 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:56,939 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:56,939 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:56,939 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:56,939 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:56,941 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,941 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,943 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:56,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:56,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:56,943 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:56,943 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:56,949 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,949 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:56,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,949 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:56,949 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,949 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,949 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,949 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:56,949 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:56,949 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,949 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:56,949 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:56,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:56,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:56,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:56,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:56,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:56,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:56,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:56,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:56,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:56,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:56,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:57,035 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:57,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39406 deadline: 1680670677033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,043 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:57,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:57,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:57,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:57,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:57,046 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:57,046 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:57,046 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:57,046 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:57,046 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,046 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:57,048 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,048 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:57,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:57,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:57,048 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,048 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:57,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:57,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:57,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:57,049 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:57,049 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:57,049 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:57,049 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:57,049 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:57,049 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:57,049 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:57,049 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:57,049 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:57,049 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:57,050 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:57,050 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:57,050 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:57,050 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:57,050 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,050 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:57,050 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:57,051 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,051 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:57,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:57,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:57,054 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,054 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:57,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:57,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:57,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:57,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:57,055 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:57,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b0c00f0[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:57,055 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,056 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,056 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:57,056 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:57,056 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:57,056 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,056 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:57,057 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:57,057 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:57,057 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,057 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:57,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:57,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:57,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:57,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:57,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:57,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:57,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:57,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:57,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:57,059 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:57,059 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:57,059 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:57,059 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:57,059 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:57,060 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:57,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:57,060 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,060 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:57,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:57,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:57,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:57,060 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:57,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:57,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:57,061 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:57,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:57,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,062 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:57,062 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:57,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:57,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:57,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:57,063 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:57,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:57,063 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:57,063 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:57,063 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:57,063 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:57,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:57,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:57,063 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:57,063 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,063 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,063 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,063 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,083 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:57,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39386 deadline: 1680670677083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,128 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/60f8c137082d4117a612352a05fd6068 2023-04-05 04:56:57,163 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:57,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:57,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:57,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:57,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:57,165 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:57,165 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:57,165 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:57,165 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:57,166 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,166 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:57,166 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,166 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:57,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:57,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:57,167 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,167 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:57,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:57,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:57,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:57,167 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:57,168 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:57,168 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:57,168 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:57,168 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:57,168 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:57,168 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:57,168 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:57,168 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:57,168 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:57,168 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:57,168 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:57,168 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:57,168 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:57,168 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,168 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:57,168 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:57,169 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,169 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:57,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:57,169 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,169 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:57,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:57,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:57,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:57,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:57,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:57,170 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:57,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@89c9dda[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:57,170 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,171 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,171 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:57,171 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:57,171 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,171 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:57,171 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:57,172 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,172 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:57,172 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:57,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,172 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:57,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:57,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:57,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:57,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:57,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:57,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:57,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:57,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:57,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:57,174 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:57,174 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:57,174 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:57,174 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:57,174 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:57,175 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,175 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,175 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:57,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:57,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:57,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:57,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:57,175 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:57,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:57,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:57,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:57,176 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:57,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:57,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,177 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,177 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:57,177 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:57,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:57,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:57,177 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:57,178 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:57,178 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:57,177 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:57,177 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:57,178 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:57,177 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:57,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:57,182 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,183 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,183 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:57,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:57,278 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:57,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:57,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:57,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:57,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:57,280 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:57,280 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:57,281 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:57,281 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:57,281 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,281 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:57,281 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,281 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:57,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:57,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:57,282 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,282 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:57,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:57,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:57,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:57,282 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:57,282 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:57,283 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:57,283 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:57,283 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:57,283 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:57,283 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:57,283 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:57,283 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:57,283 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:57,283 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:57,283 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:57,283 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:57,283 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:57,283 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,283 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:57,283 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:57,284 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,284 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:57,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:57,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:57,284 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,284 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:57,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:57,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:57,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:57,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:57,287 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:57,287 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67b3a69d[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:57,287 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,287 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:57,287 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:57,287 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,287 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:57,287 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:57,288 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,288 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:57,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:57,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:57,288 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:57,288 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:57,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:57,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:57,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:57,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:57,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:57,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:57,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:57,290 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:57,290 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:57,290 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:57,290 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:57,290 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:57,291 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,291 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,291 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:57,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:57,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:57,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,291 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:57,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:57,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:57,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:57,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:57,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:57,292 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:57,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:57,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,293 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,293 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:57,293 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:57,293 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:57,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:57,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:57,293 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:57,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:57,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:57,293 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:57,293 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:57,294 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:57,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:57,294 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:57,294 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:57,294 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,294 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,394 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:57,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:57,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:57,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:57,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:57,397 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:57,397 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:57,397 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:57,397 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:57,397 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,397 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:57,398 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:57,398 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:57,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:57,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:57,398 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,398 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:57,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:57,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:57,399 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:57,399 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:57,399 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:57,399 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:57,399 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:57,399 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:57,399 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:57,399 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:57,399 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:57,399 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:57,399 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:57,400 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:57,400 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:57,400 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:57,400 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,400 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:57,400 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:57,400 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,400 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:57,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:57,401 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,401 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:57,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:57,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:57,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:57,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:57,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:57,402 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:57,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f5acedc[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:57,402 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,403 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,403 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:57,403 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:57,403 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,403 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:57,403 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:57,404 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,404 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:57,404 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:57,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:57,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:57,404 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:57,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:57,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:57,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:57,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:57,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:57,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:57,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:57,406 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:57,406 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:57,406 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:57,406 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:57,406 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:57,407 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,407 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:57,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:57,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,407 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:57,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:57,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:57,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:57,407 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:57,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:57,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:57,409 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:57,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:57,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,411 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,411 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:57,411 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:57,411 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:57,411 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:57,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:57,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:57,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:57,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:57,411 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:57,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:57,411 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:57,411 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:57,411 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:57,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:57,411 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,411 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,411 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,511 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:57,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:57,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:57,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:57,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:57,514 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:57,514 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:57,514 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:57,514 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:57,514 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,514 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:57,515 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:57,515 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:57,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:57,515 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,515 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:57,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:57,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:57,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:57,516 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:57,516 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:57,516 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:57,516 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:57,516 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:57,516 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:57,516 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:57,516 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:57,516 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:57,516 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:57,516 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:57,516 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:57,516 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:57,516 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:57,516 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,516 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:57,516 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:57,517 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,517 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:57,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:57,517 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,517 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:57,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:57,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:57,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:57,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:57,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:57,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@442a4247[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:57,518 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:57,519 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,519 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,519 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:57,519 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:57,519 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:57,519 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,519 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:57,520 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,520 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:57,520 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:57,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:57,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:57,520 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:57,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:57,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:57,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:57,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:57,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:57,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:57,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:57,522 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:57,522 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:57,522 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:57,522 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:57,522 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:57,523 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,523 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,523 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:57,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:57,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:57,523 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:57,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:57,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:57,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:57,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:57,524 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:57,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:57,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:57,525 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,525 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:57,525 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:57,525 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:57,525 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:57,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:57,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:57,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:57,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:57,525 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:57,525 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:57,525 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:57,525 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:57,525 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:57,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:57,525 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:57,525 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,539 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/358f10e2159e4f3cb3ec8b6781ffa82a 2023-04-05 04:56:57,542 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/694443b181f34006a2a60f59779a616c as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/694443b181f34006a2a60f59779a616c 2023-04-05 04:56:57,546 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/694443b181f34006a2a60f59779a616c, entries=400, sequenceid=336, filesize=59.8 K 2023-04-05 04:56:57,547 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/60f8c137082d4117a612352a05fd6068 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/60f8c137082d4117a612352a05fd6068 2023-04-05 04:56:57,550 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/60f8c137082d4117a612352a05fd6068, entries=150, sequenceid=336, filesize=11.9 K 2023-04-05 04:56:57,551 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/358f10e2159e4f3cb3ec8b6781ffa82a as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/358f10e2159e4f3cb3ec8b6781ffa82a 2023-04-05 04:56:57,555 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/358f10e2159e4f3cb3ec8b6781ffa82a, entries=150, sequenceid=336, filesize=11.9 K 2023-04-05 04:56:57,557 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 0a496ef74db8da540124a4d88d6e37aa in 1682ms, sequenceid=336, compaction requested=true 2023-04-05 04:56:57,557 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:56:57,557 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:56:57,557 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:56:57,557 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:56:57,557 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:56:57,557 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:56:57,558 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 139904 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:56:57,558 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49256 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:56:57,558 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): 0a496ef74db8da540124a4d88d6e37aa/A is initiating minor compaction (all files) 2023-04-05 04:56:57,558 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): 0a496ef74db8da540124a4d88d6e37aa/B is initiating minor compaction (all files) 2023-04-05 04:56:57,558 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0a496ef74db8da540124a4d88d6e37aa/A in TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:57,558 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0a496ef74db8da540124a4d88d6e37aa/B in TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:57,559 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/6cde669885f94d678cc09ba2e7e894d1, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/416bec6712c043ca93221ecc364b4bd7, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/b23a9dc69e804907acd292544fdb3675, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/60f8c137082d4117a612352a05fd6068] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp, totalSize=48.1 K 2023-04-05 04:56:57,559 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/7591d7f360dd487e85d6df1a74f9e833, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/f0a81c4988e0447c8cc04bc9878208f5, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/c35a623a4df741f6a739c56823c3af80, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/694443b181f34006a2a60f59779a616c] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp, totalSize=136.6 K 2023-04-05 04:56:57,559 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 6cde669885f94d678cc09ba2e7e894d1, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1680670611436 2023-04-05 04:56:57,559 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 7591d7f360dd487e85d6df1a74f9e833, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1680670611436 2023-04-05 04:56:57,559 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 416bec6712c043ca93221ecc364b4bd7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1680670611570 2023-04-05 04:56:57,559 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting f0a81c4988e0447c8cc04bc9878208f5, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1680670611570 2023-04-05 04:56:57,560 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting b23a9dc69e804907acd292544fdb3675, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=324, earliestPutTs=1680670613706 2023-04-05 04:56:57,560 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting c35a623a4df741f6a739c56823c3af80, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=324, earliestPutTs=1680670613706 2023-04-05 04:56:57,560 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 60f8c137082d4117a612352a05fd6068, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1680670615866 2023-04-05 04:56:57,560 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 694443b181f34006a2a60f59779a616c, keycount=400, bloomtype=ROW, size=59.8 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1680670615843 2023-04-05 04:56:57,586 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): 0a496ef74db8da540124a4d88d6e37aa#B#compaction#170 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:56:57,595 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0a496ef74db8da540124a4d88d6e37aa#A#compaction#171 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:56:57,626 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:57,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:57,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:57,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:57,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:57,628 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:57,628 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:57,628 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:57,628 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:57,629 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,629 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:57,629 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:57,629 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:57,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:57,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:57,636 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:57,636 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:57,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:57,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:57,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:57,637 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:57,638 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:57,638 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:57,638 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:57,638 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:57,638 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:57,638 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:57,638 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:57,638 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:57,639 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0a496ef74db8da540124a4d88d6e37aa 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-05 04:56:57,639 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=A 2023-04-05 04:56:57,639 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:56:57,639 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=B 2023-04-05 04:56:57,639 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:56:57,639 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=C 2023-04-05 04:56:57,639 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:56:58,008 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/d1a7619e5ed14c019059be82b49b7b65 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/d1a7619e5ed14c019059be82b49b7b65 2023-04-05 04:56:58,012 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0a496ef74db8da540124a4d88d6e37aa/B of 0a496ef74db8da540124a4d88d6e37aa into d1a7619e5ed14c019059be82b49b7b65(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:56:58,012 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:56:58,012 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa., storeName=0a496ef74db8da540124a4d88d6e37aa/B, priority=12, startTime=1680670617557; duration=0sec 2023-04-05 04:56:58,012 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:56:58,012 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:56:58,014 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49256 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:56:58,014 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): 0a496ef74db8da540124a4d88d6e37aa/C is initiating minor compaction (all files) 2023-04-05 04:56:58,014 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0a496ef74db8da540124a4d88d6e37aa/C in TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:58,014 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/ecce563d265d4fe5be6cfff1aedbf142, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/1326761fd322418b964419f51e7e2b2b, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/8e078cf80e414dbd85f3c0b918e2ba71, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/358f10e2159e4f3cb3ec8b6781ffa82a] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp, totalSize=48.1 K 2023-04-05 04:56:58,014 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting ecce563d265d4fe5be6cfff1aedbf142, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1680670611436 2023-04-05 04:56:58,015 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 1326761fd322418b964419f51e7e2b2b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1680670611570 2023-04-05 04:56:58,015 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 8e078cf80e414dbd85f3c0b918e2ba71, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=324, earliestPutTs=1680670613706 2023-04-05 04:56:58,015 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 358f10e2159e4f3cb3ec8b6781ffa82a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1680670615866 2023-04-05 04:56:58,023 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): 0a496ef74db8da540124a4d88d6e37aa#C#compaction#173 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:56:58,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:58,046 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:58,061 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405a493730841b1401d92f5fec68570fbb8 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405a493730841b1401d92f5fec68570fbb8 2023-04-05 04:56:58,074 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:58,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39406 deadline: 1680670678071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:58,090 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:58,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39386 deadline: 1680670678089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:58,178 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:58,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39406 deadline: 1680670678175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:58,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:58,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39406 deadline: 1680670678379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:58,453 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/2170ee133c3d486d8447e84bfdbad223 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/2170ee133c3d486d8447e84bfdbad223 2023-04-05 04:56:58,458 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0a496ef74db8da540124a4d88d6e37aa/C of 0a496ef74db8da540124a4d88d6e37aa into 2170ee133c3d486d8447e84bfdbad223(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:56:58,458 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:56:58,458 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa., storeName=0a496ef74db8da540124a4d88d6e37aa/C, priority=12, startTime=1680670617557; duration=0sec 2023-04-05 04:56:58,458 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:56:58,472 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=372, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/2f2a2c6e1ac14c5bbc5790fddfb64127 2023-04-05 04:56:58,489 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:58,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39372 deadline: 1680670678488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:58,491 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/271439e73e0c4071bdb01067da32f2bb 2023-04-05 04:56:58,491 DEBUG [Thread-911] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8160 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa., hostname=jenkins-hbase20.apache.org,38091,1680670569206, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 04:56:58,508 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/75cc5a5c45834982a58fbf56171209a4 2023-04-05 04:56:58,512 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/2f2a2c6e1ac14c5bbc5790fddfb64127 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/2f2a2c6e1ac14c5bbc5790fddfb64127 2023-04-05 04:56:58,515 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/2f2a2c6e1ac14c5bbc5790fddfb64127, entries=150, sequenceid=372, filesize=25.4 K 2023-04-05 04:56:58,516 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/271439e73e0c4071bdb01067da32f2bb as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/271439e73e0c4071bdb01067da32f2bb 2023-04-05 04:56:58,517 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:58,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39422 deadline: 1680670678513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:58,519 DEBUG [Thread-913] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8188 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa., hostname=jenkins-hbase20.apache.org,38091,1680670569206, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 04:56:58,520 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/271439e73e0c4071bdb01067da32f2bb, entries=150, sequenceid=372, filesize=11.9 K 2023-04-05 04:56:58,521 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/75cc5a5c45834982a58fbf56171209a4 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/75cc5a5c45834982a58fbf56171209a4 2023-04-05 04:56:58,525 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/75cc5a5c45834982a58fbf56171209a4, entries=150, sequenceid=372, filesize=11.9 K 2023-04-05 04:56:58,526 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 0a496ef74db8da540124a4d88d6e37aa in 887ms, sequenceid=372, compaction requested=false 2023-04-05 04:56:58,526 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:56:58,526 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:58,526 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:58,526 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:58,526 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:58,526 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:58,526 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:58,529 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:58,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:58,529 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:58,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:58,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:58,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:58,530 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:58,530 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:58,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:58,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:58,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:58,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:58,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:58,531 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:58,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28347555[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:58,531 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:58,531 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:58,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:58,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:58,531 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:58,531 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:58,531 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:58,531 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:58,532 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:58,533 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:58,533 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:58,533 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:58,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:58,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:58,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:58,533 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:58,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:58,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:58,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:58,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:58,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:58,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:58,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:58,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:58,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:58,537 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:58,537 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:58,537 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:58,537 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:58,537 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:58,542 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:58,542 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:58,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:58,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:58,542 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:58,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:58,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:58,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:58,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:58,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:58,543 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:58,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:58,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:58,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:58,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:58,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:58,544 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:58,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:58,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:58,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:58,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:58,545 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:58,545 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:58,545 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:58,545 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:58,545 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:58,545 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:58,545 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:58,545 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:58,545 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:58,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:58,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:58,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:58,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:58,545 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:58,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:56:58,546 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:58,546 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:58,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:58,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:58,646 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:58,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:58,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:58,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:58,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:58,648 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:58,648 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:58,648 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:58,648 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:58,649 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:58,649 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:58,650 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:58,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:58,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:58,650 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:58,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:58,650 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:58,650 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:58,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:58,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:58,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:58,651 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:58,651 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:58,651 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:58,651 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:58,651 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:58,651 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:58,651 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:58,651 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:58,651 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:58,651 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:58,651 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0a496ef74db8da540124a4d88d6e37aa 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 04:56:58,651 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=A 2023-04-05 04:56:58,652 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:58,652 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=B 2023-04-05 04:56:58,652 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:58,652 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=C 2023-04-05 04:56:58,652 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:56:58,668 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405501e631b24fc4e95924a90f77e8dad1d to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405501e631b24fc4e95924a90f77e8dad1d 2023-04-05 04:56:58,674 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=385, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/fef37102d7074bc38d9f15f71f85adf7 2023-04-05 04:56:58,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:58,682 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:58,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670618717, max timestamp=1680670618731Num uniques 859; 2023-04-05 04:56:58,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670618717, max timestamp=1680670618731Num uniques 859; 2023-04-05 04:56:58,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670618717, max timestamp=1680670618731Num uniques 859; 2023-04-05 04:56:58,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:56:58,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:56:58,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:56:58,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:58,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39406 deadline: 1680670678767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:58,818 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/1c13211226d24acdaed43ef0aa54b0f1 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/1c13211226d24acdaed43ef0aa54b0f1 2023-04-05 04:56:58,823 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0a496ef74db8da540124a4d88d6e37aa/A of 0a496ef74db8da540124a4d88d6e37aa into 1c13211226d24acdaed43ef0aa54b0f1(size=26.1 K), total size for store is 51.5 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-05 04:56:58,823 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:56:58,823 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa., storeName=0a496ef74db8da540124a4d88d6e37aa/A, priority=12, startTime=1680670617557; duration=1sec 2023-04-05 04:56:58,823 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:56:58,869 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:58,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39406 deadline: 1680670678868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:59,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:59,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39406 deadline: 1680670679070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:59,087 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=385 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/1c236c6890f84aac82b18e6d1bc36ada 2023-04-05 04:56:59,098 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=385 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/f84bb6e8f44e4d20a34ebff0fc5076c2 2023-04-05 04:56:59,101 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/fef37102d7074bc38d9f15f71f85adf7 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/fef37102d7074bc38d9f15f71f85adf7 2023-04-05 04:56:59,105 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/fef37102d7074bc38d9f15f71f85adf7, entries=150, sequenceid=385, filesize=25.4 K 2023-04-05 04:56:59,105 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/1c236c6890f84aac82b18e6d1bc36ada as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/1c236c6890f84aac82b18e6d1bc36ada 2023-04-05 04:56:59,109 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/1c236c6890f84aac82b18e6d1bc36ada, entries=150, sequenceid=385, filesize=11.9 K 2023-04-05 04:56:59,110 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/f84bb6e8f44e4d20a34ebff0fc5076c2 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/f84bb6e8f44e4d20a34ebff0fc5076c2 2023-04-05 04:56:59,113 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/f84bb6e8f44e4d20a34ebff0fc5076c2, entries=150, sequenceid=385, filesize=11.9 K 2023-04-05 04:56:59,114 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 0a496ef74db8da540124a4d88d6e37aa in 463ms, sequenceid=385, compaction requested=true 2023-04-05 04:56:59,114 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:56:59,114 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:59,114 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:56:59,114 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:56:59,114 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:59,114 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:56:59,114 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:56:59,115 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:59,115 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:59,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:59,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:59,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:59,115 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:59,115 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:56:59,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:59,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:59,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:59,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:59,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:59,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:56:59,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5249b980[Count = 0] remaining members to acquire global barrier 2023-04-05 04:56:59,116 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:56:59,116 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:59,117 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:59,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:59,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:59,117 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:56:59,117 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:56:59,117 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:56:59,117 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:59,117 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:56:59,118 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:56:59,118 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:59,118 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:59,118 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:56:59,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:59,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:59,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:59,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:59,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:59,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:59,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:59,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:59,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:59,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:59,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:56:59,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:56:59,120 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:56:59,120 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:56:59,120 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:56:59,120 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:56:59,120 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:56:59,121 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:59,121 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:59,121 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:59,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:59,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:59,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:56:59,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:56:59,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:59,121 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:59,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:56:59,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:56:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:59,122 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:59,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:56:59,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:56:59,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:59,124 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:59,124 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:59,124 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:59,124 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:59,124 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:59,124 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:56:59,124 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:59,124 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:56:59,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:59,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:59,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:56:59,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:56:59,124 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:59,124 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:56:59,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:56:59,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:56:59,124 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:56:59,124 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:56:59,225 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:56:59,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:56:59,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:56:59,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:56:59,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:56:59,227 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:56:59,227 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:59,227 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:56:59,227 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:56:59,228 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:59,228 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:59,228 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:59,228 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:56:59,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:56:59,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:56:59,228 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:59,228 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:56:59,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:59,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:56:59,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:56:59,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:56:59,229 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:56:59,229 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:56:59,229 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:56:59,229 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:56:59,229 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:56:59,229 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:56:59,229 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:56:59,229 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:56:59,229 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:56:59,229 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. started... 2023-04-05 04:56:59,230 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0a496ef74db8da540124a4d88d6e37aa 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-05 04:56:59,230 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=A 2023-04-05 04:56:59,230 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:56:59,230 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=B 2023-04-05 04:56:59,230 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:56:59,230 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=C 2023-04-05 04:56:59,230 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:56:59,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:56:59,375 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. as already flushing 2023-04-05 04:56:59,392 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:59,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39406 deadline: 1680670679391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:59,494 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:59,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39406 deadline: 1680670679493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:59,641 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040599bd796872f842d5b3d19d37b1506376 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040599bd796872f842d5b3d19d37b1506376 2023-04-05 04:56:59,696 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:56:59,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39406 deadline: 1680670679696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:56:59,866 DEBUG [Thread-920] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x16505096 to 127.0.0.1:56229 2023-04-05 04:56:59,866 DEBUG [Thread-920] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:56:59,866 DEBUG [Thread-916] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c411e08 to 127.0.0.1:56229 2023-04-05 04:56:59,867 DEBUG [Thread-916] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:56:59,868 DEBUG [Thread-918] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7605c204 to 127.0.0.1:56229 2023-04-05 04:56:59,868 DEBUG [Thread-918] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:56:59,870 DEBUG [Thread-922] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x03e1990e to 127.0.0.1:56229 2023-04-05 04:56:59,870 DEBUG [Thread-922] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:57:00,001 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:00,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39406 deadline: 1680670680000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:00,047 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=421, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/4715b0cbbbac45cfaeed05851c2bdcd1 2023-04-05 04:57:00,105 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:00,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39386 deadline: 1680670680105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:00,106 DEBUG [Thread-905] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4138 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa., hostname=jenkins-hbase20.apache.org,38091,1680670569206, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 04:57:00,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:00,465 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=421 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/41ee7c5813df45a789820ba7c36a0a52 2023-04-05 04:57:00,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39398 deadline: 1680670680464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:00,469 DEBUG [Thread-907] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18212 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa., hostname=jenkins-hbase20.apache.org,38091,1680670569206, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 04:57:00,506 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:00,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39406 deadline: 1680670680505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:00,884 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=421 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/2edff8869f1d4de7933d6a6e31ccf2c7 2023-04-05 04:57:00,894 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/4715b0cbbbac45cfaeed05851c2bdcd1 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/4715b0cbbbac45cfaeed05851c2bdcd1 2023-04-05 04:57:00,900 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/4715b0cbbbac45cfaeed05851c2bdcd1, entries=150, sequenceid=421, filesize=25.4 K 2023-04-05 04:57:00,900 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/41ee7c5813df45a789820ba7c36a0a52 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/41ee7c5813df45a789820ba7c36a0a52 2023-04-05 04:57:00,904 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/41ee7c5813df45a789820ba7c36a0a52, entries=150, sequenceid=421, filesize=11.9 K 2023-04-05 04:57:00,905 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/2edff8869f1d4de7933d6a6e31ccf2c7 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/2edff8869f1d4de7933d6a6e31ccf2c7 2023-04-05 04:57:00,910 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/2edff8869f1d4de7933d6a6e31ccf2c7, entries=150, sequenceid=421, filesize=11.9 K 2023-04-05 04:57:00,910 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 0a496ef74db8da540124a4d88d6e37aa in 1680ms, sequenceid=421, compaction requested=true 2023-04-05 04:57:00,910 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:57:00,910 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:57:00,911 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:00,911 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:00,911 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:00,911 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:00,911 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:00,912 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:00,912 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:00,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:00,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:00,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:00,912 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:00,912 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:00,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:00,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:00,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:00,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:00,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:00,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:00,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b8ca13b[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:00,913 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:00,913 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:00,914 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:00,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:00,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:00,914 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:00,914 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:00,914 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:00,914 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:00,914 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:00,915 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:00,915 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:00,915 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:00,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:00,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:00,915 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:00,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:00,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:00,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:00,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:00,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:00,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:00,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:00,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:00,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:00,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:00,917 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:00,917 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:00,917 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:00,917 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:00,917 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:00,918 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:00,918 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:00,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:00,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:00,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:00,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:00,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:00,918 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:00,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:00,918 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:00,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:00,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:00,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:00,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:00,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:00,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:00,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:00,919 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:00,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:00,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:00,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:00,920 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:00,920 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:00,920 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:00,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:00,920 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:00,920 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:00,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:00,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:00,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:00,920 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:00,920 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:00,920 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:00,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:00,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:00,920 DEBUG [Thread-915] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:00,920 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:00,921 DEBUG [Thread-915] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:00,921 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:01,021 DEBUG [Thread-915] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:01,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:01,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:57:01,521 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0a496ef74db8da540124a4d88d6e37aa 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-05 04:57:01,521 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=A 2023-04-05 04:57:01,522 DEBUG [Thread-909] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x633c81ba to 127.0.0.1:56229 2023-04-05 04:57:01,522 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:01,522 DEBUG [Thread-909] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:57:01,522 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=B 2023-04-05 04:57:01,523 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:01,523 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=C 2023-04-05 04:57:01,523 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:01,947 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405dbc0e01c880845c3920238ccf72e5ae3 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405dbc0e01c880845c3920238ccf72e5ae3 2023-04-05 04:57:02,357 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=433, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/2e6a14ebc5fe415b8ce67165824ebc89 2023-04-05 04:57:02,780 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=433 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/67d0f6b8a12a46bb8b612bae59781b3f 2023-04-05 04:57:03,202 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=433 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/519ff7ea948046c8a8ee5c99f9b14ad6 2023-04-05 04:57:03,213 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/2e6a14ebc5fe415b8ce67165824ebc89 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/2e6a14ebc5fe415b8ce67165824ebc89 2023-04-05 04:57:03,217 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/2e6a14ebc5fe415b8ce67165824ebc89, entries=150, sequenceid=433, filesize=25.4 K 2023-04-05 04:57:03,217 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/67d0f6b8a12a46bb8b612bae59781b3f as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/67d0f6b8a12a46bb8b612bae59781b3f 2023-04-05 04:57:03,221 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/67d0f6b8a12a46bb8b612bae59781b3f, entries=150, sequenceid=433, filesize=11.9 K 2023-04-05 04:57:03,222 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/519ff7ea948046c8a8ee5c99f9b14ad6 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/519ff7ea948046c8a8ee5c99f9b14ad6 2023-04-05 04:57:03,225 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/519ff7ea948046c8a8ee5c99f9b14ad6, entries=150, sequenceid=433, filesize=11.9 K 2023-04-05 04:57:03,226 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 0a496ef74db8da540124a4d88d6e37aa in 1705ms, sequenceid=433, compaction requested=true 2023-04-05 04:57:03,226 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:57:03,226 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:57:03,226 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:57:03,226 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 04:57:03,226 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:57:03,226 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 04:57:03,227 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 130878 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 04:57:03,227 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): 0a496ef74db8da540124a4d88d6e37aa/A is initiating minor compaction (all files) 2023-04-05 04:57:03,227 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61578 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 04:57:03,227 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): 0a496ef74db8da540124a4d88d6e37aa/B is initiating minor compaction (all files) 2023-04-05 04:57:03,227 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0a496ef74db8da540124a4d88d6e37aa/A in TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:57:03,227 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0a496ef74db8da540124a4d88d6e37aa/B in TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:57:03,228 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/d1a7619e5ed14c019059be82b49b7b65, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/271439e73e0c4071bdb01067da32f2bb, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/1c236c6890f84aac82b18e6d1bc36ada, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/41ee7c5813df45a789820ba7c36a0a52, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/67d0f6b8a12a46bb8b612bae59781b3f] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp, totalSize=60.1 K 2023-04-05 04:57:03,228 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/1c13211226d24acdaed43ef0aa54b0f1, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/2f2a2c6e1ac14c5bbc5790fddfb64127, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/fef37102d7074bc38d9f15f71f85adf7, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/4715b0cbbbac45cfaeed05851c2bdcd1, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/2e6a14ebc5fe415b8ce67165824ebc89] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp, totalSize=127.8 K 2023-04-05 04:57:03,228 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting d1a7619e5ed14c019059be82b49b7b65, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1680670615866 2023-04-05 04:57:03,228 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 1c13211226d24acdaed43ef0aa54b0f1, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1680670615866 2023-04-05 04:57:03,228 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 271439e73e0c4071bdb01067da32f2bb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1680670615948 2023-04-05 04:57:03,228 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 2f2a2c6e1ac14c5bbc5790fddfb64127, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1680670615948 2023-04-05 04:57:03,228 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 1c236c6890f84aac82b18e6d1bc36ada, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1680670618057 2023-04-05 04:57:03,228 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting fef37102d7074bc38d9f15f71f85adf7, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1680670618057 2023-04-05 04:57:03,228 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 41ee7c5813df45a789820ba7c36a0a52, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1680670618759 2023-04-05 04:57:03,229 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 4715b0cbbbac45cfaeed05851c2bdcd1, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1680670618759 2023-04-05 04:57:03,229 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 67d0f6b8a12a46bb8b612bae59781b3f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=433, earliestPutTs=1680670619375 2023-04-05 04:57:03,229 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 2e6a14ebc5fe415b8ce67165824ebc89, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=433, earliestPutTs=1680670619375 2023-04-05 04:57:03,236 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): 0a496ef74db8da540124a4d88d6e37aa#B#compaction#185 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:57:03,240 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0a496ef74db8da540124a4d88d6e37aa#A#compaction#186 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:57:03,654 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/bdc0086ce6bb48169d9050b483d6b413 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/bdc0086ce6bb48169d9050b483d6b413 2023-04-05 04:57:03,661 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 0a496ef74db8da540124a4d88d6e37aa/B of 0a496ef74db8da540124a4d88d6e37aa into bdc0086ce6bb48169d9050b483d6b413(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:57:03,661 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:57:03,661 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa., storeName=0a496ef74db8da540124a4d88d6e37aa/B, priority=11, startTime=1680670623226; duration=0sec 2023-04-05 04:57:03,661 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:57:03,661 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 04:57:03,663 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61578 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 04:57:03,663 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): 0a496ef74db8da540124a4d88d6e37aa/C is initiating minor compaction (all files) 2023-04-05 04:57:03,663 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0a496ef74db8da540124a4d88d6e37aa/C in TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:57:03,664 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/2170ee133c3d486d8447e84bfdbad223, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/75cc5a5c45834982a58fbf56171209a4, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/f84bb6e8f44e4d20a34ebff0fc5076c2, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/2edff8869f1d4de7933d6a6e31ccf2c7, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/519ff7ea948046c8a8ee5c99f9b14ad6] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp, totalSize=60.1 K 2023-04-05 04:57:03,664 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 2170ee133c3d486d8447e84bfdbad223, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1680670615866 2023-04-05 04:57:03,664 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 75cc5a5c45834982a58fbf56171209a4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1680670615948 2023-04-05 04:57:03,665 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting f84bb6e8f44e4d20a34ebff0fc5076c2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1680670618057 2023-04-05 04:57:03,665 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 2edff8869f1d4de7933d6a6e31ccf2c7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1680670618759 2023-04-05 04:57:03,666 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 519ff7ea948046c8a8ee5c99f9b14ad6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=433, earliestPutTs=1680670619375 2023-04-05 04:57:03,677 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): 0a496ef74db8da540124a4d88d6e37aa#C#compaction#187 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:57:04,087 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/a6181a0d84264a76a556fb38363bd143 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/a6181a0d84264a76a556fb38363bd143 2023-04-05 04:57:04,091 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 0a496ef74db8da540124a4d88d6e37aa/C of 0a496ef74db8da540124a4d88d6e37aa into a6181a0d84264a76a556fb38363bd143(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:57:04,091 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:57:04,091 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa., storeName=0a496ef74db8da540124a4d88d6e37aa/C, priority=11, startTime=1680670623226; duration=0sec 2023-04-05 04:57:04,092 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:57:04,135 DEBUG [Thread-905] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b75e45b to 127.0.0.1:56229 2023-04-05 04:57:04,135 DEBUG [Thread-905] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:57:04,483 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/012c4f8138a4434dbc5250c93e80188c as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/012c4f8138a4434dbc5250c93e80188c 2023-04-05 04:57:04,489 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 0a496ef74db8da540124a4d88d6e37aa/A of 0a496ef74db8da540124a4d88d6e37aa into 012c4f8138a4434dbc5250c93e80188c(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-05 04:57:04,489 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:57:04,489 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa., storeName=0a496ef74db8da540124a4d88d6e37aa/A, priority=11, startTime=1680670623226; duration=1sec 2023-04-05 04:57:04,489 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:57:08,529 DEBUG [Thread-913] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x43a14bcf to 127.0.0.1:56229 2023-04-05 04:57:08,529 DEBUG [Thread-913] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:57:08,558 DEBUG [Thread-911] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x661b21c2 to 127.0.0.1:56229 2023-04-05 04:57:08,558 DEBUG [Thread-911] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:57:10,555 DEBUG [Thread-907] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6017755a to 127.0.0.1:56229 2023-04-05 04:57:10,555 DEBUG [Thread-907] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:57:10,555 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-05 04:57:10,555 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(394): wrote 100 2023-04-05 04:57:10,555 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(394): wrote 16 2023-04-05 04:57:10,555 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(394): wrote 110 2023-04-05 04:57:10,555 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(394): wrote 75 2023-04-05 04:57:10,555 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(394): wrote 59 2023-04-05 04:57:10,555 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-05 04:57:10,555 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(398): read 5614 2023-04-05 04:57:10,555 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(398): read 5552 2023-04-05 04:57:10,556 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-05 04:57:10,556 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(402): scanned 2409 2023-04-05 04:57:10,556 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(403): verified 7224 rows 2023-04-05 04:57:10,556 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(402): scanned 2403 2023-04-05 04:57:10,556 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(403): verified 7206 rows 2023-04-05 04:57:10,556 INFO [Listener at localhost.localdomain/34579] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-05 04:57:10,556 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x65f279d2 to 127.0.0.1:56229 2023-04-05 04:57:10,556 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:57:10,558 INFO [Listener at localhost.localdomain/34579] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-05 04:57:10,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-05 04:57:10,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-05 04:57:10,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-05 04:57:10,562 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680670630562"}]},"ts":"1680670630562"} 2023-04-05 04:57:10,563 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-05 04:57:10,575 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-05 04:57:10,576 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0a496ef74db8da540124a4d88d6e37aa, UNASSIGN}] 2023-04-05 04:57:10,577 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0a496ef74db8da540124a4d88d6e37aa, UNASSIGN 2023-04-05 04:57:10,577 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=0a496ef74db8da540124a4d88d6e37aa, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:10,577 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670630577"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680670630577"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680670630577"}]},"ts":"1680670630577"} 2023-04-05 04:57:10,579 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206}] 2023-04-05 04:57:10,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-05 04:57:10,731 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:57:10,731 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 0a496ef74db8da540124a4d88d6e37aa, disabling compactions & flushes 2023-04-05 04:57:10,731 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:57:10,731 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:57:10,731 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. after waiting 0 ms 2023-04-05 04:57:10,731 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:57:10,731 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 0a496ef74db8da540124a4d88d6e37aa 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-05 04:57:10,732 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=A 2023-04-05 04:57:10,732 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:10,732 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=B 2023-04-05 04:57:10,732 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:10,732 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0a496ef74db8da540124a4d88d6e37aa, store=C 2023-04-05 04:57:10,732 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:10,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-05 04:57:10,915 DEBUG [regionserver/jenkins-hbase20:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-05 04:57:11,159 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405acc29bac4ed84fd78f597864726a4720 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405acc29bac4ed84fd78f597864726a4720 2023-04-05 04:57:11,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-05 04:57:11,567 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=443, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/a1441d2bfd1740f0bcfa41b4796e2412 2023-04-05 04:57:11,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-05 04:57:11,987 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/aedfd2c68fc64b208a8478dd0f87e0be 2023-04-05 04:57:12,406 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/89246b2c984c4011a877cf2237f0bff0 2023-04-05 04:57:12,411 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/A/a1441d2bfd1740f0bcfa41b4796e2412 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/a1441d2bfd1740f0bcfa41b4796e2412 2023-04-05 04:57:12,416 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/a1441d2bfd1740f0bcfa41b4796e2412, entries=50, sequenceid=443, filesize=11.7 K 2023-04-05 04:57:12,417 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/B/aedfd2c68fc64b208a8478dd0f87e0be as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/aedfd2c68fc64b208a8478dd0f87e0be 2023-04-05 04:57:12,421 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/aedfd2c68fc64b208a8478dd0f87e0be, entries=50, sequenceid=443, filesize=7.1 K 2023-04-05 04:57:12,422 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/.tmp/C/89246b2c984c4011a877cf2237f0bff0 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/89246b2c984c4011a877cf2237f0bff0 2023-04-05 04:57:12,426 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/89246b2c984c4011a877cf2237f0bff0, entries=50, sequenceid=443, filesize=7.1 K 2023-04-05 04:57:12,426 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 0a496ef74db8da540124a4d88d6e37aa in 1695ms, sequenceid=443, compaction requested=false 2023-04-05 04:57:12,446 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/16131ce65c824d4696821e74e7bab041, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/3eba1ad15e8f4642baf821813b6e098c, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/a0bfc164fe254bc69ec4a03a2217a2b5, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/46255102075e4d8a82f802be2ab91ddf, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/2231dfbcfb40404c80e8afcb4efe91b1, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/6565982749d040e69836b3fc37c0ad05, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/2604088d799a4626a5262d47fcca7a83, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/cb3d50f46248404bac70b1229f52c270, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/36464768e580455581164ad48366969d, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/8b53e8a2bb9144e68198c0ef561bc21e, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/ff90dd0a7b4e45e8921d3a0b1a20b2fa, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/da76bf31651e4f37bcaf08d34f531a6f, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/1d7d840151e447fa8fe72710e4eba529, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/931e200b679b482ab38b691433368987, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/9f31d589737e4cb5882709f205a51ac4, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/7591d7f360dd487e85d6df1a74f9e833, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/f0a81c4988e0447c8cc04bc9878208f5, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/c35a623a4df741f6a739c56823c3af80, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/694443b181f34006a2a60f59779a616c, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/1c13211226d24acdaed43ef0aa54b0f1, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/2f2a2c6e1ac14c5bbc5790fddfb64127, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/fef37102d7074bc38d9f15f71f85adf7, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/4715b0cbbbac45cfaeed05851c2bdcd1, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/2e6a14ebc5fe415b8ce67165824ebc89] to archive 2023-04-05 04:57:12,448 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 04:57:12,449 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/16131ce65c824d4696821e74e7bab041 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/16131ce65c824d4696821e74e7bab041 2023-04-05 04:57:12,450 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/3eba1ad15e8f4642baf821813b6e098c to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/3eba1ad15e8f4642baf821813b6e098c 2023-04-05 04:57:12,451 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/a0bfc164fe254bc69ec4a03a2217a2b5 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/a0bfc164fe254bc69ec4a03a2217a2b5 2023-04-05 04:57:12,452 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/46255102075e4d8a82f802be2ab91ddf to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/46255102075e4d8a82f802be2ab91ddf 2023-04-05 04:57:12,453 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/2231dfbcfb40404c80e8afcb4efe91b1 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/2231dfbcfb40404c80e8afcb4efe91b1 2023-04-05 04:57:12,454 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/6565982749d040e69836b3fc37c0ad05 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/6565982749d040e69836b3fc37c0ad05 2023-04-05 04:57:12,455 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/2604088d799a4626a5262d47fcca7a83 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/2604088d799a4626a5262d47fcca7a83 2023-04-05 04:57:12,456 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/cb3d50f46248404bac70b1229f52c270 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/cb3d50f46248404bac70b1229f52c270 2023-04-05 04:57:12,456 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/36464768e580455581164ad48366969d to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/36464768e580455581164ad48366969d 2023-04-05 04:57:12,457 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/8b53e8a2bb9144e68198c0ef561bc21e to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/8b53e8a2bb9144e68198c0ef561bc21e 2023-04-05 04:57:12,458 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/ff90dd0a7b4e45e8921d3a0b1a20b2fa to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/ff90dd0a7b4e45e8921d3a0b1a20b2fa 2023-04-05 04:57:12,459 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/da76bf31651e4f37bcaf08d34f531a6f to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/da76bf31651e4f37bcaf08d34f531a6f 2023-04-05 04:57:12,460 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/1d7d840151e447fa8fe72710e4eba529 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/1d7d840151e447fa8fe72710e4eba529 2023-04-05 04:57:12,461 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/931e200b679b482ab38b691433368987 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/931e200b679b482ab38b691433368987 2023-04-05 04:57:12,462 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/9f31d589737e4cb5882709f205a51ac4 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/9f31d589737e4cb5882709f205a51ac4 2023-04-05 04:57:12,463 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/7591d7f360dd487e85d6df1a74f9e833 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/7591d7f360dd487e85d6df1a74f9e833 2023-04-05 04:57:12,464 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/f0a81c4988e0447c8cc04bc9878208f5 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/f0a81c4988e0447c8cc04bc9878208f5 2023-04-05 04:57:12,465 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/c35a623a4df741f6a739c56823c3af80 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/c35a623a4df741f6a739c56823c3af80 2023-04-05 04:57:12,466 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/694443b181f34006a2a60f59779a616c to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/694443b181f34006a2a60f59779a616c 2023-04-05 04:57:12,467 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/1c13211226d24acdaed43ef0aa54b0f1 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/1c13211226d24acdaed43ef0aa54b0f1 2023-04-05 04:57:12,468 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/2f2a2c6e1ac14c5bbc5790fddfb64127 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/2f2a2c6e1ac14c5bbc5790fddfb64127 2023-04-05 04:57:12,468 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/fef37102d7074bc38d9f15f71f85adf7 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/fef37102d7074bc38d9f15f71f85adf7 2023-04-05 04:57:12,469 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/4715b0cbbbac45cfaeed05851c2bdcd1 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/4715b0cbbbac45cfaeed05851c2bdcd1 2023-04-05 04:57:12,470 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/2e6a14ebc5fe415b8ce67165824ebc89 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/2e6a14ebc5fe415b8ce67165824ebc89 2023-04-05 04:57:12,472 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/d59e02b1a0a5495aaf6d2692223b9578, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/c63ba82ba4e9406fafc66190f7dd50b7, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/238cbfa302c2412884b247b71efbbbbc, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/9b0cf2a048f14847ba39afe51d58b0d7, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/43a7d26cbd744889bbc61859eeec8003, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/f191857faca74ddc8baae6cf23c40e89, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/78a13d97b8d840bca3fbc69c706c30e6, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/d2d9e490fdb24f1abd1f1e836db84078, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/f676873c28724ed9a538b9ab64ac871f, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/1e9c46f4bcf34d76af03314ee7766493, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/a051711b31ce44c98c704efc4c1e4cda, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/62c08d8e072e44239c8de07c4e3f6e88, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/fbeeb3e026794f89a4cbdf7a2eef7a66, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/0bddf914895941afb72a53e6eb170c09, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/6cde669885f94d678cc09ba2e7e894d1, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/e8f783f3e4db494b8bc89d16fc98b9d5, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/416bec6712c043ca93221ecc364b4bd7, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/b23a9dc69e804907acd292544fdb3675, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/d1a7619e5ed14c019059be82b49b7b65, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/60f8c137082d4117a612352a05fd6068, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/271439e73e0c4071bdb01067da32f2bb, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/1c236c6890f84aac82b18e6d1bc36ada, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/41ee7c5813df45a789820ba7c36a0a52, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/67d0f6b8a12a46bb8b612bae59781b3f] to archive 2023-04-05 04:57:12,473 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 04:57:12,474 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/d59e02b1a0a5495aaf6d2692223b9578 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/d59e02b1a0a5495aaf6d2692223b9578 2023-04-05 04:57:12,475 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/c63ba82ba4e9406fafc66190f7dd50b7 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/c63ba82ba4e9406fafc66190f7dd50b7 2023-04-05 04:57:12,475 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/238cbfa302c2412884b247b71efbbbbc to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/238cbfa302c2412884b247b71efbbbbc 2023-04-05 04:57:12,476 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/9b0cf2a048f14847ba39afe51d58b0d7 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/9b0cf2a048f14847ba39afe51d58b0d7 2023-04-05 04:57:12,477 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/43a7d26cbd744889bbc61859eeec8003 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/43a7d26cbd744889bbc61859eeec8003 2023-04-05 04:57:12,478 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/f191857faca74ddc8baae6cf23c40e89 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/f191857faca74ddc8baae6cf23c40e89 2023-04-05 04:57:12,479 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/78a13d97b8d840bca3fbc69c706c30e6 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/78a13d97b8d840bca3fbc69c706c30e6 2023-04-05 04:57:12,479 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/d2d9e490fdb24f1abd1f1e836db84078 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/d2d9e490fdb24f1abd1f1e836db84078 2023-04-05 04:57:12,480 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/f676873c28724ed9a538b9ab64ac871f to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/f676873c28724ed9a538b9ab64ac871f 2023-04-05 04:57:12,481 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/1e9c46f4bcf34d76af03314ee7766493 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/1e9c46f4bcf34d76af03314ee7766493 2023-04-05 04:57:12,482 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/a051711b31ce44c98c704efc4c1e4cda to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/a051711b31ce44c98c704efc4c1e4cda 2023-04-05 04:57:12,483 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/62c08d8e072e44239c8de07c4e3f6e88 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/62c08d8e072e44239c8de07c4e3f6e88 2023-04-05 04:57:12,484 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/fbeeb3e026794f89a4cbdf7a2eef7a66 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/fbeeb3e026794f89a4cbdf7a2eef7a66 2023-04-05 04:57:12,484 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/0bddf914895941afb72a53e6eb170c09 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/0bddf914895941afb72a53e6eb170c09 2023-04-05 04:57:12,485 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/6cde669885f94d678cc09ba2e7e894d1 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/6cde669885f94d678cc09ba2e7e894d1 2023-04-05 04:57:12,486 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/e8f783f3e4db494b8bc89d16fc98b9d5 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/e8f783f3e4db494b8bc89d16fc98b9d5 2023-04-05 04:57:12,487 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/416bec6712c043ca93221ecc364b4bd7 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/416bec6712c043ca93221ecc364b4bd7 2023-04-05 04:57:12,488 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/b23a9dc69e804907acd292544fdb3675 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/b23a9dc69e804907acd292544fdb3675 2023-04-05 04:57:12,489 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/d1a7619e5ed14c019059be82b49b7b65 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/d1a7619e5ed14c019059be82b49b7b65 2023-04-05 04:57:12,490 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/60f8c137082d4117a612352a05fd6068 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/60f8c137082d4117a612352a05fd6068 2023-04-05 04:57:12,491 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/271439e73e0c4071bdb01067da32f2bb to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/271439e73e0c4071bdb01067da32f2bb 2023-04-05 04:57:12,492 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/1c236c6890f84aac82b18e6d1bc36ada to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/1c236c6890f84aac82b18e6d1bc36ada 2023-04-05 04:57:12,493 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/41ee7c5813df45a789820ba7c36a0a52 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/41ee7c5813df45a789820ba7c36a0a52 2023-04-05 04:57:12,494 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/67d0f6b8a12a46bb8b612bae59781b3f to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/67d0f6b8a12a46bb8b612bae59781b3f 2023-04-05 04:57:12,497 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/e4b7263161bf4f06972ebc0ab5509095, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/127aef55aed246c98c9814a5a804604d, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/286d8a278e7d49b7be61ea2129c4e59c, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/077e1e0832f84261b1b78b6990e361ba, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/07cecf3d8bb7404eac56dca98661dc9a, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/19c08e36c73444e58a2684a11148ed88, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/89cc91b1a22a4b11895f83b5b00a5eb7, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/95a16fa9862949898af4c62528b39c85, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/c36e47081a37470e96c0c5d0c0588a45, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/f7d64b92a0aa4465ba21225ad9278315, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/dcc201d119cd4e74b64f050a90e45f98, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/77531efe7c1a40658c837c4fc3e458ea, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/ba47dea7ed2b4fa2960d21e610211d10, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/c61fec32fc0c4ab9aaf7623e50a3151e, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/ecce563d265d4fe5be6cfff1aedbf142, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/1e2a79dd37d04e13b5625d918f7965db, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/1326761fd322418b964419f51e7e2b2b, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/8e078cf80e414dbd85f3c0b918e2ba71, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/2170ee133c3d486d8447e84bfdbad223, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/358f10e2159e4f3cb3ec8b6781ffa82a, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/75cc5a5c45834982a58fbf56171209a4, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/f84bb6e8f44e4d20a34ebff0fc5076c2, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/2edff8869f1d4de7933d6a6e31ccf2c7, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/519ff7ea948046c8a8ee5c99f9b14ad6] to archive 2023-04-05 04:57:12,498 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 04:57:12,499 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/e4b7263161bf4f06972ebc0ab5509095 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/e4b7263161bf4f06972ebc0ab5509095 2023-04-05 04:57:12,500 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/127aef55aed246c98c9814a5a804604d to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/127aef55aed246c98c9814a5a804604d 2023-04-05 04:57:12,501 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/286d8a278e7d49b7be61ea2129c4e59c to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/286d8a278e7d49b7be61ea2129c4e59c 2023-04-05 04:57:12,501 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/077e1e0832f84261b1b78b6990e361ba to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/077e1e0832f84261b1b78b6990e361ba 2023-04-05 04:57:12,502 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/07cecf3d8bb7404eac56dca98661dc9a to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/07cecf3d8bb7404eac56dca98661dc9a 2023-04-05 04:57:12,503 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/19c08e36c73444e58a2684a11148ed88 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/19c08e36c73444e58a2684a11148ed88 2023-04-05 04:57:12,504 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/89cc91b1a22a4b11895f83b5b00a5eb7 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/89cc91b1a22a4b11895f83b5b00a5eb7 2023-04-05 04:57:12,504 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/95a16fa9862949898af4c62528b39c85 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/95a16fa9862949898af4c62528b39c85 2023-04-05 04:57:12,505 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/c36e47081a37470e96c0c5d0c0588a45 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/c36e47081a37470e96c0c5d0c0588a45 2023-04-05 04:57:12,506 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/f7d64b92a0aa4465ba21225ad9278315 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/f7d64b92a0aa4465ba21225ad9278315 2023-04-05 04:57:12,506 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/dcc201d119cd4e74b64f050a90e45f98 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/dcc201d119cd4e74b64f050a90e45f98 2023-04-05 04:57:12,507 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/77531efe7c1a40658c837c4fc3e458ea to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/77531efe7c1a40658c837c4fc3e458ea 2023-04-05 04:57:12,508 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/ba47dea7ed2b4fa2960d21e610211d10 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/ba47dea7ed2b4fa2960d21e610211d10 2023-04-05 04:57:12,508 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/c61fec32fc0c4ab9aaf7623e50a3151e to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/c61fec32fc0c4ab9aaf7623e50a3151e 2023-04-05 04:57:12,509 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/ecce563d265d4fe5be6cfff1aedbf142 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/ecce563d265d4fe5be6cfff1aedbf142 2023-04-05 04:57:12,510 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/1e2a79dd37d04e13b5625d918f7965db to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/1e2a79dd37d04e13b5625d918f7965db 2023-04-05 04:57:12,511 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/1326761fd322418b964419f51e7e2b2b to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/1326761fd322418b964419f51e7e2b2b 2023-04-05 04:57:12,512 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/8e078cf80e414dbd85f3c0b918e2ba71 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/8e078cf80e414dbd85f3c0b918e2ba71 2023-04-05 04:57:12,513 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/2170ee133c3d486d8447e84bfdbad223 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/2170ee133c3d486d8447e84bfdbad223 2023-04-05 04:57:12,514 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/358f10e2159e4f3cb3ec8b6781ffa82a to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/358f10e2159e4f3cb3ec8b6781ffa82a 2023-04-05 04:57:12,515 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/75cc5a5c45834982a58fbf56171209a4 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/75cc5a5c45834982a58fbf56171209a4 2023-04-05 04:57:12,516 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/f84bb6e8f44e4d20a34ebff0fc5076c2 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/f84bb6e8f44e4d20a34ebff0fc5076c2 2023-04-05 04:57:12,517 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/2edff8869f1d4de7933d6a6e31ccf2c7 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/2edff8869f1d4de7933d6a6e31ccf2c7 2023-04-05 04:57:12,518 DEBUG [StoreCloser-TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/519ff7ea948046c8a8ee5c99f9b14ad6 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/519ff7ea948046c8a8ee5c99f9b14ad6 2023-04-05 04:57:12,523 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/recovered.edits/446.seqid, newMaxSeqId=446, maxSeqId=4 2023-04-05 04:57:12,525 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa. 2023-04-05 04:57:12,525 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 0a496ef74db8da540124a4d88d6e37aa: 2023-04-05 04:57:12,527 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:57:12,527 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=0a496ef74db8da540124a4d88d6e37aa, regionState=CLOSED 2023-04-05 04:57:12,528 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670632527"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680670632527"}]},"ts":"1680670632527"} 2023-04-05 04:57:12,530 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-05 04:57:12,530 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 0a496ef74db8da540124a4d88d6e37aa, server=jenkins-hbase20.apache.org,38091,1680670569206 in 1.9500 sec 2023-04-05 04:57:12,532 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-05 04:57:12,532 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=0a496ef74db8da540124a4d88d6e37aa, UNASSIGN in 1.9540 sec 2023-04-05 04:57:12,533 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680670632533"}]},"ts":"1680670632533"} 2023-04-05 04:57:12,534 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-05 04:57:12,535 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-05 04:57:12,537 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9770 sec 2023-04-05 04:57:12,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-05 04:57:12,670 INFO [Listener at localhost.localdomain/34579] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-05 04:57:12,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-05 04:57:12,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 04:57:12,675 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 04:57:12,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-05 04:57:12,677 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 04:57:12,684 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:57:12,688 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A, FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B, FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C, FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/recovered.edits] 2023-04-05 04:57:12,690 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/012c4f8138a4434dbc5250c93e80188c to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/012c4f8138a4434dbc5250c93e80188c 2023-04-05 04:57:12,691 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/a1441d2bfd1740f0bcfa41b4796e2412 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/A/a1441d2bfd1740f0bcfa41b4796e2412 2023-04-05 04:57:12,693 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/aedfd2c68fc64b208a8478dd0f87e0be to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/aedfd2c68fc64b208a8478dd0f87e0be 2023-04-05 04:57:12,694 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/bdc0086ce6bb48169d9050b483d6b413 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/B/bdc0086ce6bb48169d9050b483d6b413 2023-04-05 04:57:12,696 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/89246b2c984c4011a877cf2237f0bff0 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/89246b2c984c4011a877cf2237f0bff0 2023-04-05 04:57:12,697 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/a6181a0d84264a76a556fb38363bd143 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/C/a6181a0d84264a76a556fb38363bd143 2023-04-05 04:57:12,700 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/recovered.edits/446.seqid to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa/recovered.edits/446.seqid 2023-04-05 04:57:12,701 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/0a496ef74db8da540124a4d88d6e37aa 2023-04-05 04:57:12,701 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-05 04:57:12,702 DEBUG [PEWorker-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-05 04:57:12,703 DEBUG [PEWorker-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-05 04:57:12,707 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304053322db6bb18b41689165133622788a9a to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304053322db6bb18b41689165133622788a9a 2023-04-05 04:57:12,708 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304053732d9dc097846c08fa2e91aef136ee1 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304053732d9dc097846c08fa2e91aef136ee1 2023-04-05 04:57:12,710 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040542c712f9ef0c4f549e6e04fd06993e40 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040542c712f9ef0c4f549e6e04fd06993e40 2023-04-05 04:57:12,711 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405440a5e36ba88401297676503ad457b42 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405440a5e36ba88401297676503ad457b42 2023-04-05 04:57:12,713 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405501e631b24fc4e95924a90f77e8dad1d to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405501e631b24fc4e95924a90f77e8dad1d 2023-04-05 04:57:12,715 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405631a013f702d459cb03dd5e0c78200af to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405631a013f702d459cb03dd5e0c78200af 2023-04-05 04:57:12,717 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304056a2d20dacdf44ca88e4d223087c40054 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304056a2d20dacdf44ca88e4d223087c40054 2023-04-05 04:57:12,718 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304056d4875fa382d48839b90e8162af9ea35 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304056d4875fa382d48839b90e8162af9ea35 2023-04-05 04:57:12,719 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040587ffdfc611374dee8a4f61597dfe445b to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040587ffdfc611374dee8a4f61597dfe445b 2023-04-05 04:57:12,720 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304058e8d43be1b6240bba53a0ccbde9625bf to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304058e8d43be1b6240bba53a0ccbde9625bf 2023-04-05 04:57:12,722 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040591a2f7958d1b42cd9cbeed43f3b134a9 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040591a2f7958d1b42cd9cbeed43f3b134a9 2023-04-05 04:57:12,723 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040599bd796872f842d5b3d19d37b1506376 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040599bd796872f842d5b3d19d37b1506376 2023-04-05 04:57:12,724 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405a493730841b1401d92f5fec68570fbb8 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405a493730841b1401d92f5fec68570fbb8 2023-04-05 04:57:12,725 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405acc29bac4ed84fd78f597864726a4720 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405acc29bac4ed84fd78f597864726a4720 2023-04-05 04:57:12,726 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405bd4d96c135404ba0be98eeaf62e52ffa to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405bd4d96c135404ba0be98eeaf62e52ffa 2023-04-05 04:57:12,727 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405ca71227a2bee405697cbe0015e110692 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405ca71227a2bee405697cbe0015e110692 2023-04-05 04:57:12,728 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405cc6f7b77da5248dcaf8a69e3f3dbc3a4 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405cc6f7b77da5248dcaf8a69e3f3dbc3a4 2023-04-05 04:57:12,729 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405cd461254d87a41b3a67b014b80fa1118 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405cd461254d87a41b3a67b014b80fa1118 2023-04-05 04:57:12,730 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405d4aa31a9784a4295af42b3fb27dedf45 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405d4aa31a9784a4295af42b3fb27dedf45 2023-04-05 04:57:12,731 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405dbc0e01c880845c3920238ccf72e5ae3 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405dbc0e01c880845c3920238ccf72e5ae3 2023-04-05 04:57:12,732 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405ee7383dc59974a94b0e8eec9c821a39a to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405ee7383dc59974a94b0e8eec9c821a39a 2023-04-05 04:57:12,732 DEBUG [PEWorker-2] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-05 04:57:12,734 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 04:57:12,735 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-05 04:57:12,737 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-05 04:57:12,738 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 04:57:12,738 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-05 04:57:12,738 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680670632738"}]},"ts":"9223372036854775807"} 2023-04-05 04:57:12,739 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-05 04:57:12,740 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 0a496ef74db8da540124a4d88d6e37aa, NAME => 'TestAcidGuarantees,,1680670596666.0a496ef74db8da540124a4d88d6e37aa.', STARTKEY => '', ENDKEY => ''}] 2023-04-05 04:57:12,740 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-05 04:57:12,740 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680670632740"}]},"ts":"9223372036854775807"} 2023-04-05 04:57:12,741 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-05 04:57:12,742 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 04:57:12,743 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 71 msec 2023-04-05 04:57:12,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-05 04:57:12,778 INFO [Listener at localhost.localdomain/34579] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-05 04:57:12,792 INFO [Listener at localhost.localdomain/34579] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=227 (was 237), OpenFileDescriptor=509 (was 581), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=287 (was 324), ProcessCount=166 (was 168), AvailableMemoryMB=6384 (was 6518) 2023-04-05 04:57:12,802 INFO [Listener at localhost.localdomain/34579] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=227, OpenFileDescriptor=509, MaxFileDescriptor=60000, SystemLoadAverage=287, ProcessCount=166, AvailableMemoryMB=6384 2023-04-05 04:57:12,804 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-05 04:57:12,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-05 04:57:12,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-05 04:57:12,807 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-05 04:57:12,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-05 04:57:12,808 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-05 04:57:12,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-05 04:57:12,809 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a 2023-04-05 04:57:12,810 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a empty. 2023-04-05 04:57:12,810 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a 2023-04-05 04:57:12,810 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-05 04:57:12,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-05 04:57:13,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-05 04:57:13,222 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-05 04:57:13,227 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => d690627297fb3f579eb1d6c5d71ead3a, NAME => 'TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp 2023-04-05 04:57:13,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-05 04:57:13,636 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 04:57:13,637 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing d690627297fb3f579eb1d6c5d71ead3a, disabling compactions & flushes 2023-04-05 04:57:13,637 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:13,637 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:13,637 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. after waiting 0 ms 2023-04-05 04:57:13,637 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:13,637 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:13,637 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:13,643 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-05 04:57:13,645 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670633645"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680670633645"}]},"ts":"1680670633645"} 2023-04-05 04:57:13,647 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-05 04:57:13,647 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-05 04:57:13,648 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680670633648"}]},"ts":"1680670633648"} 2023-04-05 04:57:13,649 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-05 04:57:13,651 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d690627297fb3f579eb1d6c5d71ead3a, ASSIGN}] 2023-04-05 04:57:13,653 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d690627297fb3f579eb1d6c5d71ead3a, ASSIGN 2023-04-05 04:57:13,654 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=d690627297fb3f579eb1d6c5d71ead3a, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,38091,1680670569206; forceNewPlan=false, retain=false 2023-04-05 04:57:13,806 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=d690627297fb3f579eb1d6c5d71ead3a, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:13,807 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670633806"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680670633806"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680670633806"}]},"ts":"1680670633806"} 2023-04-05 04:57:13,810 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206}] 2023-04-05 04:57:13,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-05 04:57:13,968 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:13,969 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d690627297fb3f579eb1d6c5d71ead3a, NAME => 'TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.', STARTKEY => '', ENDKEY => ''} 2023-04-05 04:57:13,969 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees d690627297fb3f579eb1d6c5d71ead3a 2023-04-05 04:57:13,969 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 04:57:13,969 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for d690627297fb3f579eb1d6c5d71ead3a 2023-04-05 04:57:13,969 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for d690627297fb3f579eb1d6c5d71ead3a 2023-04-05 04:57:13,971 INFO [StoreOpener-d690627297fb3f579eb1d6c5d71ead3a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region d690627297fb3f579eb1d6c5d71ead3a 2023-04-05 04:57:13,974 DEBUG [StoreOpener-d690627297fb3f579eb1d6c5d71ead3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A 2023-04-05 04:57:13,974 DEBUG [StoreOpener-d690627297fb3f579eb1d6c5d71ead3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A 2023-04-05 04:57:13,974 INFO [StoreOpener-d690627297fb3f579eb1d6c5d71ead3a-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 04:57:13,974 INFO [StoreOpener-d690627297fb3f579eb1d6c5d71ead3a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d690627297fb3f579eb1d6c5d71ead3a columnFamilyName A 2023-04-05 04:57:13,975 INFO [StoreOpener-d690627297fb3f579eb1d6c5d71ead3a-1] regionserver.HStore(310): Store=d690627297fb3f579eb1d6c5d71ead3a/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 04:57:13,975 INFO [StoreOpener-d690627297fb3f579eb1d6c5d71ead3a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region d690627297fb3f579eb1d6c5d71ead3a 2023-04-05 04:57:13,977 DEBUG [StoreOpener-d690627297fb3f579eb1d6c5d71ead3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B 2023-04-05 04:57:13,977 DEBUG [StoreOpener-d690627297fb3f579eb1d6c5d71ead3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B 2023-04-05 04:57:13,977 INFO [StoreOpener-d690627297fb3f579eb1d6c5d71ead3a-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 04:57:13,977 INFO [StoreOpener-d690627297fb3f579eb1d6c5d71ead3a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d690627297fb3f579eb1d6c5d71ead3a columnFamilyName B 2023-04-05 04:57:13,978 INFO [StoreOpener-d690627297fb3f579eb1d6c5d71ead3a-1] regionserver.HStore(310): Store=d690627297fb3f579eb1d6c5d71ead3a/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 04:57:13,978 INFO [StoreOpener-d690627297fb3f579eb1d6c5d71ead3a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region d690627297fb3f579eb1d6c5d71ead3a 2023-04-05 04:57:13,980 DEBUG [StoreOpener-d690627297fb3f579eb1d6c5d71ead3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C 2023-04-05 04:57:13,980 DEBUG [StoreOpener-d690627297fb3f579eb1d6c5d71ead3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C 2023-04-05 04:57:13,980 INFO [StoreOpener-d690627297fb3f579eb1d6c5d71ead3a-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 04:57:13,980 INFO [StoreOpener-d690627297fb3f579eb1d6c5d71ead3a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d690627297fb3f579eb1d6c5d71ead3a columnFamilyName C 2023-04-05 04:57:13,981 INFO [StoreOpener-d690627297fb3f579eb1d6c5d71ead3a-1] regionserver.HStore(310): Store=d690627297fb3f579eb1d6c5d71ead3a/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 04:57:13,981 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:13,982 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a 2023-04-05 04:57:13,982 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a 2023-04-05 04:57:13,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-05 04:57:13,987 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for d690627297fb3f579eb1d6c5d71ead3a 2023-04-05 04:57:13,988 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-05 04:57:13,989 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened d690627297fb3f579eb1d6c5d71ead3a; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71282715, jitterRate=0.062195226550102234}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-05 04:57:13,989 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:13,989 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a., pid=30, masterSystemTime=1680670633964 2023-04-05 04:57:13,991 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:13,991 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:13,991 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=d690627297fb3f579eb1d6c5d71ead3a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:13,991 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670633991"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680670633991"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680670633991"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680670633991"}]},"ts":"1680670633991"} 2023-04-05 04:57:13,993 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-05 04:57:13,993 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 in 182 msec 2023-04-05 04:57:13,994 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-05 04:57:13,994 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=d690627297fb3f579eb1d6c5d71ead3a, ASSIGN in 342 msec 2023-04-05 04:57:13,994 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-05 04:57:13,995 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680670633994"}]},"ts":"1680670633994"} 2023-04-05 04:57:13,995 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-05 04:57:13,997 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-05 04:57:13,998 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1930 sec 2023-04-05 04:57:14,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-05 04:57:14,918 INFO [Listener at localhost.localdomain/34579] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-05 04:57:14,920 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x4d3f4b11 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:57:14,927 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c466b6c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:57:14,928 DEBUG [Listener at localhost.localdomain/34579] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:57:14,930 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41410, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:57:14,931 DEBUG [Listener at localhost.localdomain/34579] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-05 04:57:14,933 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57112, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-05 04:57:14,935 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x7187fcf3 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:57:14,942 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@46d74611, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:57:14,943 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x712d78ca to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:57:14,950 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3f837278, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:57:14,951 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x407b9cb0 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:57:14,954 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a8a575d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:57:14,954 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x7f456fa2 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:57:14,957 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6347f2e6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:57:14,958 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x62be7a0b to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:57:14,961 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@36523181, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:57:14,962 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x6b458e37 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:57:14,975 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4cbc74e7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:57:14,975 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x7bae1a64 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:57:14,984 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3c5d1710, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:57:14,984 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x5654f8ca to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:57:14,991 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5d22a55e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:57:14,992 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x509adaa8 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:57:15,001 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2179ead9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:57:15,001 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x7e5e7a93 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:57:15,009 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7897b06a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:57:15,015 DEBUG [hconnection-0x221b1640-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:57:15,015 DEBUG [hconnection-0x51b7880-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:57:15,015 DEBUG [hconnection-0xd1e76fa-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:57:15,015 DEBUG [hconnection-0x40723217-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:57:15,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:15,016 DEBUG [hconnection-0x3e67f21d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:57:15,016 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41432, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:57:15,018 DEBUG [hconnection-0x34935a9b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:57:15,018 DEBUG [hconnection-0x7ad124b4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:57:15,026 DEBUG [hconnection-0x5f1b0e8c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:57:15,026 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41420, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:57:15,026 DEBUG [hconnection-0xd4bf2f5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:57:15,026 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41430, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:57:15,026 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41434, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:57:15,026 DEBUG [hconnection-0x2936220f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:57:15,028 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41456, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:57:15,028 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41444, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:57:15,028 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41454, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:57:15,029 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41482, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:57:15,031 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41486, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:57:15,035 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41488, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:57:15,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:15,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:15,042 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:15,042 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:15,042 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:15,042 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:15,043 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,043 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:15,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:15,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:15,043 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:15,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:15,044 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,044 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:15,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:15,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:15,045 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:15,046 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:15,048 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:15,048 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:15,048 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:15,048 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:15,052 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:15,052 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:15,052 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:15,053 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:15,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on d690627297fb3f579eb1d6c5d71ead3a 2023-04-05 04:57:15,053 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d690627297fb3f579eb1d6c5d71ead3a 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 04:57:15,053 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:15,053 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:15,053 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:15,054 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:15,054 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,054 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:15,054 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:15,057 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=A 2023-04-05 04:57:15,057 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:15,057 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=B 2023-04-05 04:57:15,058 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:15,058 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=C 2023-04-05 04:57:15,058 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:15,059 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,059 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:15,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:15,059 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,059 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:15,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:15,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:15,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:15,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:15,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2da2e477[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:15,062 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:15,062 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,066 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,066 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:15,066 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,066 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:15,066 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:15,066 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:15,067 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:15,067 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,067 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:15,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:15,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:15,067 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:15,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:15,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:15,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:15,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:15,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:15,069 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:15,070 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:15,070 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:15,070 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:15,070 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:15,071 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,071 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,071 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:15,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:15,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:15,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:15,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:15,071 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:15,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:15,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,072 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:15,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,073 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,073 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:15,073 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:15,073 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:15,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:15,074 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:15,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:15,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:15,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:15,073 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:15,073 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:15,074 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,074 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,074 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:15,074 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:15,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:15,074 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:15,086 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670635071, max timestamp=1680670635083Num uniques 859; 2023-04-05 04:57:15,086 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670635071, max timestamp=1680670635083Num uniques 859; 2023-04-05 04:57:15,086 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:57:15,086 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670635071, max timestamp=1680670635083Num uniques 859; 2023-04-05 04:57:15,086 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:57:15,086 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:57:15,138 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:15,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41420 deadline: 1680670695096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,142 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:15,142 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:15,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41454 deadline: 1680670695138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1680670695138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:15,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41430 deadline: 1680670695138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,143 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:15,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41488 deadline: 1680670695139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,174 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:15,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:15,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:15,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:15,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:15,178 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:15,178 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:15,178 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:15,178 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:15,178 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,178 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:15,179 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:15,179 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:15,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:15,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:15,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:15,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:15,180 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,180 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:15,180 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:15,182 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:15,182 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:15,183 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:15,183 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:15,183 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:15,185 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:15,185 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:15,185 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:15,185 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:15,185 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:15,185 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:15,185 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:15,185 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:15,185 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,185 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:15,185 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:15,191 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,191 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:15,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:15,192 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,192 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:15,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:15,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:15,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:15,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:15,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26b7a24b[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:15,193 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:15,193 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,200 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,200 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:15,200 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:15,200 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:15,200 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,200 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:15,207 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,208 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:15,208 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:15,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:15,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:15,208 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:15,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:15,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:15,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:15,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:15,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:15,212 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:15,212 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:15,212 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:15,212 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:15,212 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:15,216 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,216 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,217 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:15,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,217 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:15,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:15,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:15,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:15,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:15,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,219 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:15,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:15,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,225 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,225 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:15,225 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:15,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:15,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:15,225 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:15,225 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:15,225 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:15,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:15,225 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:15,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:15,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:15,225 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:15,225 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:15,226 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:15,225 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,226 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,241 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:15,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41420 deadline: 1680670695239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:15,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1680670695243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:15,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41454 deadline: 1680670695243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:15,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41488 deadline: 1680670695244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,246 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:15,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41430 deadline: 1680670695244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,326 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:15,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:15,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:15,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:15,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:15,328 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:15,328 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:15,328 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:15,328 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:15,329 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,329 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:15,329 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,329 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:15,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:15,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:15,330 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,330 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:15,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:15,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:15,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:15,330 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:15,330 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:15,331 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:15,331 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:15,331 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:15,331 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:15,331 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:15,331 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:15,331 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:15,331 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:15,331 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:15,331 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:15,331 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:15,331 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:15,331 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,331 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:15,331 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:15,332 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:15,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:15,332 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:15,332 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,332 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:15,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:15,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:15,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:15,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39ad8aa4[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:15,333 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:15,334 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,334 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,334 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:15,334 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:15,334 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:15,342 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,342 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:15,350 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:15,350 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,350 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:15,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,350 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:15,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:15,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:15,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:15,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:15,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:15,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:15,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:15,352 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:15,352 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:15,352 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:15,352 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:15,352 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:15,353 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,353 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:15,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:15,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,353 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:15,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:15,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:15,353 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:15,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:15,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:15,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,355 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,356 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,356 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:15,356 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:15,356 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:15,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:15,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:15,357 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,356 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:15,357 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:15,357 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,357 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:15,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:15,357 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:15,357 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:15,357 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:15,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:15,443 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:15,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41420 deadline: 1680670695442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,447 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:15,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41454 deadline: 1680670695446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,448 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:15,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41488 deadline: 1680670695446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,448 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:15,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1680670695447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:15,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41430 deadline: 1680670695447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,457 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:15,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:15,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:15,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:15,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:15,459 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:15,459 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:15,460 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:15,460 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:15,460 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,460 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:15,460 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:15,460 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:15,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:15,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:15,461 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,461 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:15,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:15,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:15,461 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:15,462 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:15,462 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:15,462 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:15,462 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:15,462 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:15,462 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:15,462 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:15,462 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:15,462 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:15,462 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:15,462 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:15,462 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:15,462 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:15,462 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,462 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:15,462 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:15,463 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,463 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:15,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:15,463 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,463 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:15,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:15,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:15,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:15,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:15,465 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:15,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@677671d3[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:15,465 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,466 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,466 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:15,466 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:15,466 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:15,466 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,466 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:15,467 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,467 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:15,467 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:15,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:15,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:15,467 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:15,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:15,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:15,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:15,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:15,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:15,469 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:15,469 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:15,469 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:15,469 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:15,469 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:15,470 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,470 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,470 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:15,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:15,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:15,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:15,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:15,471 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:15,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:15,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,472 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:15,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,473 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:15,473 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:15,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:15,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:15,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:15,473 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:15,473 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:15,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:15,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:15,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:15,474 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:15,474 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:15,474 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:15,474 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,474 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,474 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,474 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,474 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:15,492 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/e03765a3b5fa4ab7a2934d24ba02a64e 2023-04-05 04:57:15,533 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/f93c90844a7f4b52933ee0214ed0a06f 2023-04-05 04:57:15,552 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/a7eacc04f5204c82977f172a667413d5 2023-04-05 04:57:15,557 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/e03765a3b5fa4ab7a2934d24ba02a64e as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/e03765a3b5fa4ab7a2934d24ba02a64e 2023-04-05 04:57:15,561 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/e03765a3b5fa4ab7a2934d24ba02a64e, entries=150, sequenceid=13, filesize=11.6 K 2023-04-05 04:57:15,561 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/f93c90844a7f4b52933ee0214ed0a06f as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/f93c90844a7f4b52933ee0214ed0a06f 2023-04-05 04:57:15,565 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/f93c90844a7f4b52933ee0214ed0a06f, entries=150, sequenceid=13, filesize=11.6 K 2023-04-05 04:57:15,566 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/a7eacc04f5204c82977f172a667413d5 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/a7eacc04f5204c82977f172a667413d5 2023-04-05 04:57:15,572 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/a7eacc04f5204c82977f172a667413d5, entries=150, sequenceid=13, filesize=11.6 K 2023-04-05 04:57:15,573 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=127.47 KB/130530 for d690627297fb3f579eb1d6c5d71ead3a in 520ms, sequenceid=13, compaction requested=false 2023-04-05 04:57:15,573 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:15,574 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:15,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:15,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:15,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:15,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:15,577 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:15,577 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:15,577 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:15,577 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:15,577 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,577 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:15,578 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:15,578 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:15,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:15,578 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,578 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:15,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:15,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:15,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:15,579 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:15,579 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:15,579 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:15,580 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:15,580 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:15,580 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:15,580 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:15,580 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:15,580 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:15,580 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:15,580 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d690627297fb3f579eb1d6c5d71ead3a 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-05 04:57:15,580 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=A 2023-04-05 04:57:15,580 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:15,580 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=B 2023-04-05 04:57:15,580 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:15,580 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=C 2023-04-05 04:57:15,581 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:15,612 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/e3a9273c3384408ba281d860bfbc81dd 2023-04-05 04:57:15,628 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/13fefedd2c76485bbf3b3b484d1c81c0 2023-04-05 04:57:15,644 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/e8711c202c63409987a5888a3935781d 2023-04-05 04:57:15,648 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/e3a9273c3384408ba281d860bfbc81dd as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/e3a9273c3384408ba281d860bfbc81dd 2023-04-05 04:57:15,651 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/e3a9273c3384408ba281d860bfbc81dd, entries=150, sequenceid=48, filesize=11.6 K 2023-04-05 04:57:15,652 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/13fefedd2c76485bbf3b3b484d1c81c0 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/13fefedd2c76485bbf3b3b484d1c81c0 2023-04-05 04:57:15,656 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/13fefedd2c76485bbf3b3b484d1c81c0, entries=150, sequenceid=48, filesize=11.6 K 2023-04-05 04:57:15,657 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/e8711c202c63409987a5888a3935781d as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/e8711c202c63409987a5888a3935781d 2023-04-05 04:57:15,666 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/e8711c202c63409987a5888a3935781d, entries=150, sequenceid=48, filesize=11.6 K 2023-04-05 04:57:15,667 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=0 B/0 for d690627297fb3f579eb1d6c5d71ead3a in 87ms, sequenceid=48, compaction requested=false 2023-04-05 04:57:15,667 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:15,667 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:15,667 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:15,667 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:15,667 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,667 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:15,667 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:15,668 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:15,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:15,668 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,669 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,669 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:15,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:15,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:15,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:15,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:15,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dcda016[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:15,670 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:15,670 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,671 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,671 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:15,671 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:15,671 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:15,672 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,672 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:15,672 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:15,672 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:15,672 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,672 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:15,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:15,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:15,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:15,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:15,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:15,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:15,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:15,675 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:15,675 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:15,675 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:15,676 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:15,676 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:15,676 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,676 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,676 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:15,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:15,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:15,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,677 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:15,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:15,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:15,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:15,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,679 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:15,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,683 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:15,683 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:15,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:15,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:15,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:15,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:15,684 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,684 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:15,684 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:15,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:15,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:15,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:15,684 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,684 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:15,684 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:15,684 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:15,684 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,685 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,685 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on d690627297fb3f579eb1d6c5d71ead3a 2023-04-05 04:57:15,751 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d690627297fb3f579eb1d6c5d71ead3a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 04:57:15,752 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=A 2023-04-05 04:57:15,752 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:15,752 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=B 2023-04-05 04:57:15,752 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:15,752 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=C 2023-04-05 04:57:15,752 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:15,784 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:15,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:15,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:15,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:15,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:15,787 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:15,787 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:15,787 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:15,787 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:15,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670635767, max timestamp=1680670635785Num uniques 859; 2023-04-05 04:57:15,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670635767, max timestamp=1680670635786Num uniques 859; 2023-04-05 04:57:15,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:57:15,788 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,788 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:15,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670635767, max timestamp=1680670635785Num uniques 859; 2023-04-05 04:57:15,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:57:15,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:57:15,788 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:15,789 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:15,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:15,789 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,789 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:15,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:15,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:15,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:15,790 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:15,790 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:15,790 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:15,790 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:15,790 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:15,790 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:15,791 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:15,791 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:15,791 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:15,791 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:15,791 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:15,791 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:15,791 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:15,791 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:15,791 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,791 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:15,791 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:15,792 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,792 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:15,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:15,792 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,792 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:15,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:15,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:15,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:15,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:15,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4673f41e[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:15,793 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:15,793 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,794 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,794 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:15,794 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:15,794 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:15,794 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,794 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:15,795 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,795 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:15,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:15,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:15,795 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:15,795 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:15,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:15,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:15,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:15,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:15,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:15,796 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:15,796 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:15,796 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:15,796 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:15,796 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:15,797 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,797 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:15,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:15,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,797 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:15,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:15,798 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:15,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:15,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:15,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,799 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:15,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,800 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,800 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:15,800 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:15,800 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:15,800 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:15,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:15,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:15,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:15,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:15,800 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:15,800 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,800 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,800 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,800 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:15,800 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:15,801 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:15,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:15,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:15,802 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:15,802 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:15,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41488 deadline: 1680670695798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1680670695799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:15,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41430 deadline: 1680670695801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:15,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41454 deadline: 1680670695802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,805 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:15,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41420 deadline: 1680670695802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,901 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:15,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:15,906 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:15,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:15,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41430 deadline: 1680670695903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,906 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:15,906 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:15,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41454 deadline: 1680670695905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41488 deadline: 1680670695906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,907 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:15,907 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:15,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1680670695906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41420 deadline: 1680670695906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:15,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:15,908 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:15,908 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:15,908 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:15,908 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:15,908 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,908 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:15,909 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:15,909 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:15,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:15,909 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,909 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:15,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:15,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:15,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:15,910 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:15,910 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:15,910 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:15,910 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:15,910 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:15,910 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:15,911 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:15,911 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:15,911 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:15,911 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:15,911 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:15,911 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:15,911 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:15,911 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:15,911 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,911 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:15,911 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:15,912 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,912 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:15,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:15,912 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,912 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:15,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:15,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:15,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:15,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13deaede[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:15,913 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:15,913 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,914 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,914 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:15,914 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:15,914 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:15,914 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,914 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:15,916 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:15,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:15,916 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:15,916 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:15,916 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:15,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:15,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:15,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:15,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:15,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:15,918 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:15,918 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:15,918 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:15,918 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:15,918 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:15,919 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,919 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:15,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,920 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:15,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:15,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:15,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:15,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:15,920 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:15,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,922 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:15,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:15,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,923 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:15,923 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:15,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:15,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:15,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:15,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:15,924 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,924 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:15,924 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:15,924 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:15,924 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,924 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:15,924 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:15,924 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:15,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:15,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:15,925 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:15,925 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:16,025 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:16,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:16,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:16,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:16,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:16,028 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:16,028 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:16,028 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:16,028 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:16,028 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,028 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:16,029 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,029 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:16,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:16,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:16,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:16,029 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,029 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:16,030 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:16,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:16,030 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:16,030 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:16,030 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:16,030 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:16,030 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:16,030 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:16,031 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:16,031 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:16,031 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:16,031 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:16,031 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:16,031 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:16,031 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:16,031 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:16,031 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,031 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:16,031 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:16,032 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:16,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:16,032 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,032 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,032 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:16,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:16,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:16,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:16,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:16,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:16,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47f2bd9f[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:16,034 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:16,034 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,034 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,035 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:16,035 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:16,035 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:16,035 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,035 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:16,035 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:16,035 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:16,035 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,035 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:16,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:16,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:16,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:16,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:16,038 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:16,038 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:16,038 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:16,038 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:16,038 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:16,039 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,039 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:16,039 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:16,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:16,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:16,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:16,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:16,040 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:16,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:16,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:16,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,041 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:16,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:16,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:16,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:16,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:16,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:16,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:16,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,043 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:16,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:16,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:16,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:16,043 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:16,044 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:16,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:16,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:16,110 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:16,110 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:16,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41454 deadline: 1680670696107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41430 deadline: 1680670696107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,110 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:16,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41420 deadline: 1680670696108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,110 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:16,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41488 deadline: 1680670696108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,110 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:16,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1680670696109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,144 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:16,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:16,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:16,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:16,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:16,146 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:16,146 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:16,147 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:16,147 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:16,147 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,147 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:16,148 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:16,148 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:16,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:16,148 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,148 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:16,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:16,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:16,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:16,148 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:16,149 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:16,149 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:16,149 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:16,149 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:16,149 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:16,149 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:16,149 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:16,149 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:16,149 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:16,149 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:16,149 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:16,149 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:16,149 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:16,149 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,149 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:16,149 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:16,150 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,150 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:16,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:16,150 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,150 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:16,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:16,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:16,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:16,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:16,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:16,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@770f7867[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:16,152 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:16,152 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,152 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,152 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:16,152 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:16,152 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,152 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:16,152 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:16,153 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,153 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:16,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:16,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:16,153 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:16,153 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:16,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:16,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:16,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:16,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:16,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:16,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:16,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:16,155 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:16,155 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:16,155 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:16,155 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:16,155 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:16,156 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,156 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,156 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:16,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:16,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:16,156 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:16,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:16,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:16,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:16,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:16,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:16,157 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:16,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:16,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,159 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,159 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:16,159 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:16,159 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:16,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:16,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:16,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:16,159 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:16,159 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:16,159 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,159 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:16,159 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:16,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:16,159 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,159 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:16,159 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:16,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:16,172 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/71b1752120ff4768b9da68b665bfdf0c 2023-04-05 04:57:16,259 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:16,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:16,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:16,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:16,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:16,262 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:16,262 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:16,262 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:16,262 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:16,262 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,262 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:16,263 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,263 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:16,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:16,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:16,263 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,263 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:16,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:16,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:16,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:16,264 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:16,264 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:16,264 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:16,264 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:16,264 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:16,264 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:16,264 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:16,264 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:16,264 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:16,264 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:16,265 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:16,265 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:16,265 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:16,265 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:16,265 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,265 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:16,265 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:16,266 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,266 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:16,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:16,266 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,266 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:16,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:16,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:16,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:16,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:16,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:16,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e05b19c[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:16,267 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:16,267 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,268 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,268 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:16,268 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:16,268 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,268 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:16,268 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:16,269 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,269 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:16,269 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:16,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,269 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:16,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:16,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:16,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:16,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:16,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:16,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:16,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:16,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:16,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:16,271 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:16,271 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:16,271 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:16,271 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:16,271 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:16,272 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,272 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:16,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:16,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,272 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:16,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:16,272 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:16,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:16,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:16,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:16,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:16,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:16,273 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:16,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,275 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,276 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:16,276 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:16,276 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,276 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:16,276 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:16,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:16,276 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:16,276 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:16,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:16,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:16,276 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,276 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:16,276 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,276 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:16,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:16,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:16,376 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:16,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:16,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:16,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:16,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:16,379 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:16,379 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:16,379 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:16,379 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:16,379 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,379 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:16,380 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:16,380 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:16,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:16,380 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,380 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:16,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:16,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:16,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:16,381 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:16,381 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:16,381 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:16,381 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:16,381 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:16,381 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:16,381 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:16,381 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:16,381 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:16,381 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:16,381 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:16,382 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:16,382 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:16,382 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:16,382 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,382 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:16,382 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:16,382 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,382 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:16,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:16,383 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,383 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:16,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:16,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:16,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:16,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:16,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:16,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@484b42dc[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:16,384 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:16,384 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,384 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,384 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:16,384 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:16,384 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,385 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:16,385 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:16,385 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,385 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:16,385 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:16,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:16,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:16,385 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:16,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:16,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:16,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:16,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:16,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:16,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:16,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:16,387 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:16,387 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:16,387 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:16,387 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:16,387 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:16,388 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,388 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:16,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:16,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,388 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:16,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:16,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:16,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:16,389 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:16,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:16,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:16,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,390 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:16,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:16,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,391 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,391 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:16,391 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:16,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:16,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:16,392 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:16,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:16,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:16,392 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:16,392 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,392 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,392 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,392 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:16,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:16,392 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:16,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:16,392 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:16,392 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:16,411 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:16,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41420 deadline: 1680670696411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:16,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1680670696411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,414 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:16,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41488 deadline: 1680670696412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,414 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:16,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41454 deadline: 1680670696413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,414 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:16,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41430 deadline: 1680670696413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,493 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:16,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:16,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:16,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:16,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:16,495 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:16,495 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:16,495 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:16,495 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:16,496 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,496 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:16,496 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,496 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:16,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:16,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:16,496 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,496 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:16,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:16,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:16,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:16,497 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:16,497 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:16,497 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:16,497 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:16,497 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:16,497 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:16,498 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:16,498 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:16,498 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:16,498 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:16,498 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:16,498 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:16,498 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:16,498 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:16,498 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,498 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:16,498 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:16,499 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,499 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:16,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:16,499 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,499 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:16,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:16,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:16,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:16,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:16,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:16,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a07000f[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:16,500 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:16,500 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,500 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,500 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:16,501 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:16,501 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:16,501 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,501 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:16,501 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,501 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:16,502 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:16,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:16,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:16,502 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:16,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:16,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:16,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:16,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:16,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:16,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:16,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:16,503 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:16,503 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:16,503 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:16,503 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:16,503 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:16,504 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,504 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:16,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:16,504 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:16,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:16,505 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:16,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:16,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:16,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:16,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:16,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,506 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:16,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:16,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,507 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,507 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:16,507 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:16,507 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:16,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:16,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:16,507 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:16,507 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:16,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:16,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:16,507 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:16,507 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:16,507 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,508 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:16,508 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:16,508 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:16,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:16,582 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/baf1242aeb6a488ebcd64227638117b1 2023-04-05 04:57:16,593 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/064dce90bec24c8e887fb7a138f164ed 2023-04-05 04:57:16,596 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/71b1752120ff4768b9da68b665bfdf0c as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/71b1752120ff4768b9da68b665bfdf0c 2023-04-05 04:57:16,599 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/71b1752120ff4768b9da68b665bfdf0c, entries=100, sequenceid=59, filesize=9.3 K 2023-04-05 04:57:16,600 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/baf1242aeb6a488ebcd64227638117b1 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/baf1242aeb6a488ebcd64227638117b1 2023-04-05 04:57:16,605 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/baf1242aeb6a488ebcd64227638117b1, entries=100, sequenceid=59, filesize=9.3 K 2023-04-05 04:57:16,606 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/064dce90bec24c8e887fb7a138f164ed as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/064dce90bec24c8e887fb7a138f164ed 2023-04-05 04:57:16,608 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:16,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:16,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:16,610 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/064dce90bec24c8e887fb7a138f164ed, entries=100, sequenceid=59, filesize=9.3 K 2023-04-05 04:57:16,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:16,611 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for d690627297fb3f579eb1d6c5d71ead3a in 860ms, sequenceid=59, compaction requested=true 2023-04-05 04:57:16,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:16,611 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:16,611 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:16,611 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:16,611 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:16,619 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:57:16,619 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:57:16,619 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:16,619 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:57:16,619 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:57:16,619 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:57:16,619 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,619 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:16,621 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:57:16,621 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:57:16,621 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): d690627297fb3f579eb1d6c5d71ead3a/A is initiating minor compaction (all files) 2023-04-05 04:57:16,621 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): d690627297fb3f579eb1d6c5d71ead3a/B is initiating minor compaction (all files) 2023-04-05 04:57:16,621 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:16,621 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d690627297fb3f579eb1d6c5d71ead3a/A in TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:16,621 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d690627297fb3f579eb1d6c5d71ead3a/B in TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:16,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:16,621 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/f93c90844a7f4b52933ee0214ed0a06f, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/13fefedd2c76485bbf3b3b484d1c81c0, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/baf1242aeb6a488ebcd64227638117b1] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp, totalSize=32.5 K 2023-04-05 04:57:16,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:16,621 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/e03765a3b5fa4ab7a2934d24ba02a64e, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/e3a9273c3384408ba281d860bfbc81dd, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/71b1752120ff4768b9da68b665bfdf0c] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp, totalSize=32.5 K 2023-04-05 04:57:16,622 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting f93c90844a7f4b52933ee0214ed0a06f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680670635049 2023-04-05 04:57:16,622 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:16,622 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting e03765a3b5fa4ab7a2934d24ba02a64e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680670635049 2023-04-05 04:57:16,622 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,622 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:16,622 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 13fefedd2c76485bbf3b3b484d1c81c0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680670635099 2023-04-05 04:57:16,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:16,623 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting e3a9273c3384408ba281d860bfbc81dd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680670635099 2023-04-05 04:57:16,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:16,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:16,623 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:16,623 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting baf1242aeb6a488ebcd64227638117b1, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1680670635748 2023-04-05 04:57:16,623 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 71b1752120ff4768b9da68b665bfdf0c, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1680670635748 2023-04-05 04:57:16,623 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:16,623 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:16,623 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:16,623 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:16,624 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:16,624 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:16,624 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:16,624 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:16,624 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:16,624 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d690627297fb3f579eb1d6c5d71ead3a 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-05 04:57:16,624 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=A 2023-04-05 04:57:16,624 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:16,624 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=B 2023-04-05 04:57:16,624 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:16,624 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=C 2023-04-05 04:57:16,624 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:16,639 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): d690627297fb3f579eb1d6c5d71ead3a#B#compaction#201 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:57:16,643 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): d690627297fb3f579eb1d6c5d71ead3a#A#compaction#202 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:57:16,650 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/0fe53263e7e64c38aa4c8b363f1d4d89 2023-04-05 04:57:16,657 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/5898d13b365c42379357b107a35c0ae5 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/5898d13b365c42379357b107a35c0ae5 2023-04-05 04:57:16,662 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d690627297fb3f579eb1d6c5d71ead3a/A of d690627297fb3f579eb1d6c5d71ead3a into 5898d13b365c42379357b107a35c0ae5(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:57:16,662 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:16,662 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a., storeName=d690627297fb3f579eb1d6c5d71ead3a/A, priority=13, startTime=1680670636611; duration=0sec 2023-04-05 04:57:16,662 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:57:16,662 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:57:16,663 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:57:16,663 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): d690627297fb3f579eb1d6c5d71ead3a/C is initiating minor compaction (all files) 2023-04-05 04:57:16,663 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d690627297fb3f579eb1d6c5d71ead3a/C in TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:16,664 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/a7eacc04f5204c82977f172a667413d5, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/e8711c202c63409987a5888a3935781d, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/064dce90bec24c8e887fb7a138f164ed] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp, totalSize=32.5 K 2023-04-05 04:57:16,664 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting a7eacc04f5204c82977f172a667413d5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680670635049 2023-04-05 04:57:16,664 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting e8711c202c63409987a5888a3935781d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680670635099 2023-04-05 04:57:16,664 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 064dce90bec24c8e887fb7a138f164ed, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1680670635748 2023-04-05 04:57:16,669 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): d690627297fb3f579eb1d6c5d71ead3a#C#compaction#204 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:57:16,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on d690627297fb3f579eb1d6c5d71ead3a 2023-04-05 04:57:16,916 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:16,921 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:16,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41488 deadline: 1680670696919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:16,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41420 deadline: 1680670696920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:16,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1680670696920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:16,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41430 deadline: 1680670696921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:16,924 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:16,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41454 deadline: 1680670696922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,026 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:17,027 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:17,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41420 deadline: 1680670697024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1680670697024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,027 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:17,027 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:17,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41430 deadline: 1680670697024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41454 deadline: 1680670697024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,049 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/95e6c856f88345e6b41243056b76d376 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/95e6c856f88345e6b41243056b76d376 2023-04-05 04:57:17,052 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d690627297fb3f579eb1d6c5d71ead3a/B of d690627297fb3f579eb1d6c5d71ead3a into 95e6c856f88345e6b41243056b76d376(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:57:17,052 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:17,053 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a., storeName=d690627297fb3f579eb1d6c5d71ead3a/B, priority=13, startTime=1680670636619; duration=0sec 2023-04-05 04:57:17,053 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:57:17,061 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/0dce95b61fca4325863fbb8812756c3f 2023-04-05 04:57:17,078 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/fdcdce7652c84cd2ad63738fd66c27bf as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/fdcdce7652c84cd2ad63738fd66c27bf 2023-04-05 04:57:17,082 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d690627297fb3f579eb1d6c5d71ead3a/C of d690627297fb3f579eb1d6c5d71ead3a into fdcdce7652c84cd2ad63738fd66c27bf(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:57:17,082 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:17,082 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a., storeName=d690627297fb3f579eb1d6c5d71ead3a/C, priority=13, startTime=1680670636619; duration=0sec 2023-04-05 04:57:17,082 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:57:17,229 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:17,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41420 deadline: 1680670697228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,230 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:17,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1680670697228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,230 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:17,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41430 deadline: 1680670697229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,230 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:17,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41454 deadline: 1680670697229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,470 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/4902aed2c06a4511864e5c1ed071ee78 2023-04-05 04:57:17,474 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/0fe53263e7e64c38aa4c8b363f1d4d89 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/0fe53263e7e64c38aa4c8b363f1d4d89 2023-04-05 04:57:17,478 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/0fe53263e7e64c38aa4c8b363f1d4d89, entries=150, sequenceid=96, filesize=11.6 K 2023-04-05 04:57:17,481 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/0dce95b61fca4325863fbb8812756c3f as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/0dce95b61fca4325863fbb8812756c3f 2023-04-05 04:57:17,489 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/0dce95b61fca4325863fbb8812756c3f, entries=150, sequenceid=96, filesize=11.6 K 2023-04-05 04:57:17,489 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-05 04:57:17,490 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/4902aed2c06a4511864e5c1ed071ee78 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/4902aed2c06a4511864e5c1ed071ee78 2023-04-05 04:57:17,495 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/4902aed2c06a4511864e5c1ed071ee78, entries=150, sequenceid=96, filesize=11.6 K 2023-04-05 04:57:17,496 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for d690627297fb3f579eb1d6c5d71ead3a in 872ms, sequenceid=96, compaction requested=false 2023-04-05 04:57:17,496 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:17,496 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:17,496 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:17,496 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:17,496 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,496 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:17,496 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:17,497 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,497 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:17,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:17,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:17,498 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:17,498 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:17,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:17,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:17,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:17,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:17,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:17,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16479f68[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:17,506 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:17,507 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:17,516 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:17,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:17,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:17,516 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:17,516 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:17,516 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:17,517 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,517 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:17,518 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:17,518 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:17,518 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:17,518 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:17,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:17,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:17,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:17,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:17,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:17,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:17,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:17,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:17,522 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:17,522 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:17,522 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:17,522 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:17,522 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:17,523 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:17,523 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:17,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:17,523 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:17,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:17,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:17,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:17,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:17,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:17,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:17,524 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:17,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:17,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:17,527 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:17,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:17,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:17,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:17,530 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,530 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:17,530 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:17,531 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:17,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:17,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:17,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:17,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:17,531 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:17,531 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:17,530 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:17,531 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:17,531 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,531 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:17,531 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:17,531 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:17,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:17,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:17,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:17,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on d690627297fb3f579eb1d6c5d71ead3a 2023-04-05 04:57:17,534 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d690627297fb3f579eb1d6c5d71ead3a 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 04:57:17,535 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=A 2023-04-05 04:57:17,535 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:17,535 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=B 2023-04-05 04:57:17,535 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:17,535 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=C 2023-04-05 04:57:17,535 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:17,562 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680670637553, max timestamp=1680670637560Num uniques 859; 2023-04-05 04:57:17,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1680670637553, max timestamp=1680670637560Num uniques 859; 2023-04-05 04:57:17,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 04:57:17,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 04:57:17,564 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680670637553, max timestamp=1680670637560Num uniques 859; 2023-04-05 04:57:17,564 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 04:57:17,571 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=110 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/7482240a5d7248229e208d1009e062d7 2023-04-05 04:57:17,581 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=110 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/43b3a9e64b594fd2a3232f1416f0b992 2023-04-05 04:57:17,582 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:17,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41420 deadline: 1680670697578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,585 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:17,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1680670697580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,585 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:17,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41430 deadline: 1680670697582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:17,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41454 deadline: 1680670697582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,631 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:17,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:17,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:17,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:17,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:17,634 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:17,634 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:17,634 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:17,634 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:17,635 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:17,635 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:17,636 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:17,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:17,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:17,636 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:17,637 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,637 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:17,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:17,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:17,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:17,637 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:17,637 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:17,637 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:17,637 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:17,638 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:17,638 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:17,640 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:17,640 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:17,642 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:17,642 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:17,642 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:17,642 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:17,642 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:17,642 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:17,642 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,642 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:17,642 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:17,650 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,650 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:17,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:17,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:17,650 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:17,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:17,650 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:17,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:17,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:17,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:17,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:17,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15cd456d[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:17,652 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:17,652 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:17,658 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:17,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:17,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:17,658 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:17,658 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:17,658 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,658 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:17,658 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:17,659 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:17,659 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,659 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:17,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,659 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:17,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:17,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:17,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:17,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:17,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:17,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:17,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:17,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:17,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:17,671 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:17,671 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:17,671 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:17,671 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:17,671 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:17,674 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:17,674 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:17,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:17,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:17,674 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:17,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:17,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:17,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:17,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:17,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:17,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:17,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:17,675 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:17,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:17,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:17,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,676 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:17,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:17,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,678 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,678 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:17,678 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:17,678 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:17,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:17,678 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:17,678 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:17,678 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:17,678 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:17,678 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:17,678 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:17,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:17,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:17,678 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:17,678 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:17,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:17,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:17,684 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:17,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41420 deadline: 1680670697683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,687 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:17,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1680670697686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,688 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:17,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41430 deadline: 1680670697686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,688 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:17,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41454 deadline: 1680670697687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,778 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:17,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:17,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:17,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:17,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:17,781 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:17,781 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:17,781 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:17,781 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:17,782 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:17,782 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:17,782 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,782 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:17,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:17,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:17,782 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,782 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:17,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:17,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:17,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:17,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:17,783 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:17,784 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:17,784 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:17,784 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:17,784 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:17,784 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:17,784 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:17,784 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:17,784 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:17,784 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:17,784 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:17,784 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:17,784 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:17,784 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:17,784 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,784 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:17,784 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:17,786 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:17,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:17,786 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:17,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:17,788 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:17,788 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:17,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:17,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:17,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:17,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:17,789 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:17,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@702b8aa[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:17,789 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:17,790 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:17,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:17,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:17,790 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:17,790 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:17,790 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:17,790 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,790 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:17,790 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,790 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:17,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:17,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:17,790 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:17,791 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:17,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:17,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:17,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:17,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:17,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:17,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:17,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:17,792 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:17,792 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:17,792 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:17,792 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:17,793 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:17,793 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:17,793 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:17,793 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:17,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:17,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:17,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:17,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:17,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:17,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:17,794 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:17,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:17,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:17,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:17,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:17,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:17,795 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:17,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:17,797 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:17,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:17,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:17,797 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,798 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:17,798 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:17,798 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,798 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:17,798 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:17,798 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:17,798 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:17,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:17,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:17,798 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:17,798 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:17,798 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:17,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:17,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:17,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:17,886 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:17,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41420 deadline: 1680670697885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,891 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:17,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41430 deadline: 1680670697889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,891 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:17,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1680670697890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,891 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:17,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41454 deadline: 1680670697890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,899 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:17,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:17,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:17,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:17,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:17,911 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:17,911 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:17,911 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:17,911 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:17,912 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:17,912 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:17,913 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:17,913 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:17,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:17,913 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,913 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:17,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:17,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:17,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:17,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:17,914 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:17,918 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:17,918 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:17,918 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:17,918 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:17,918 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:17,926 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:17,926 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:17,926 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:17,926 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:17,926 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:17,926 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:17,927 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:17,927 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:17,927 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,927 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:17,927 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:17,928 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,928 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:17,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:17,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:17,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:17,929 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:17,929 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:17,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:17,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:17,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:17,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:17,932 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:17,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25a7d73d[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:17,932 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:17,932 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:17,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41488 deadline: 1680670697931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,933 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:17,933 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,933 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:17,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:17,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:17,933 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:17,933 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:17,933 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:17,934 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,934 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:17,934 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:17,934 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:17,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:17,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:17,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:17,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:17,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:17,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:17,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:17,938 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:17,938 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:17,938 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:17,938 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:17,938 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:17,941 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:17,941 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:17,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:17,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:17,942 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:17,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:17,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:17,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:17,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:17,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:17,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:17,942 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:17,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:17,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:17,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:17,944 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:17,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:17,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,946 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:17,946 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:17,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:17,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:17,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:17,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:17,946 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,947 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:17,947 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:17,947 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:17,947 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:17,947 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:17,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:17,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:17,947 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:17,947 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:17,947 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:17,947 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:17,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:18,020 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=110 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/764ba28c608f4718895e8cf8217945bd 2023-04-05 04:57:18,024 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/7482240a5d7248229e208d1009e062d7 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/7482240a5d7248229e208d1009e062d7 2023-04-05 04:57:18,027 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/7482240a5d7248229e208d1009e062d7, entries=150, sequenceid=110, filesize=11.6 K 2023-04-05 04:57:18,029 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/43b3a9e64b594fd2a3232f1416f0b992 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/43b3a9e64b594fd2a3232f1416f0b992 2023-04-05 04:57:18,034 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/43b3a9e64b594fd2a3232f1416f0b992, entries=150, sequenceid=110, filesize=11.6 K 2023-04-05 04:57:18,037 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/764ba28c608f4718895e8cf8217945bd as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/764ba28c608f4718895e8cf8217945bd 2023-04-05 04:57:18,042 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/764ba28c608f4718895e8cf8217945bd, entries=150, sequenceid=110, filesize=11.6 K 2023-04-05 04:57:18,043 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for d690627297fb3f579eb1d6c5d71ead3a in 508ms, sequenceid=110, compaction requested=true 2023-04-05 04:57:18,043 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:18,043 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:57:18,045 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:57:18,045 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:57:18,045 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:57:18,045 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:57:18,046 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:57:18,046 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): d690627297fb3f579eb1d6c5d71ead3a/A is initiating minor compaction (all files) 2023-04-05 04:57:18,046 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d690627297fb3f579eb1d6c5d71ead3a/A in TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:18,046 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/5898d13b365c42379357b107a35c0ae5, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/0fe53263e7e64c38aa4c8b363f1d4d89, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/7482240a5d7248229e208d1009e062d7] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp, totalSize=34.9 K 2023-04-05 04:57:18,046 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:57:18,046 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): d690627297fb3f579eb1d6c5d71ead3a/B is initiating minor compaction (all files) 2023-04-05 04:57:18,046 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d690627297fb3f579eb1d6c5d71ead3a/B in TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:18,046 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/95e6c856f88345e6b41243056b76d376, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/0dce95b61fca4325863fbb8812756c3f, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/43b3a9e64b594fd2a3232f1416f0b992] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp, totalSize=34.9 K 2023-04-05 04:57:18,046 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 5898d13b365c42379357b107a35c0ae5, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1680670635099 2023-04-05 04:57:18,047 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 95e6c856f88345e6b41243056b76d376, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1680670635099 2023-04-05 04:57:18,047 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 0fe53263e7e64c38aa4c8b363f1d4d89, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1680670635795 2023-04-05 04:57:18,047 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 0dce95b61fca4325863fbb8812756c3f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1680670635795 2023-04-05 04:57:18,047 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:18,050 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 7482240a5d7248229e208d1009e062d7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1680670636920 2023-04-05 04:57:18,050 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 43b3a9e64b594fd2a3232f1416f0b992, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1680670636920 2023-04-05 04:57:18,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:18,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:18,057 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): d690627297fb3f579eb1d6c5d71ead3a#A#compaction#209 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:57:18,060 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-05 04:57:18,063 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): d690627297fb3f579eb1d6c5d71ead3a#B#compaction#210 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:57:18,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:18,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:18,074 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:18,074 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:18,074 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:18,074 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:18,075 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:18,075 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:18,076 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:18,077 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:18,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:18,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:18,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:18,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:18,078 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:18,078 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:18,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:18,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:18,078 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:18,090 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:18,090 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:18,090 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:18,090 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:18,090 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:18,102 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:18,102 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:18,112 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/9923316413824b0e807ef015f987c70f as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/9923316413824b0e807ef015f987c70f 2023-04-05 04:57:18,118 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:18,118 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:18,118 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d690627297fb3f579eb1d6c5d71ead3a 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-05 04:57:18,120 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=A 2023-04-05 04:57:18,120 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:18,121 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=B 2023-04-05 04:57:18,122 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:18,122 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=C 2023-04-05 04:57:18,122 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:18,124 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d690627297fb3f579eb1d6c5d71ead3a/A of d690627297fb3f579eb1d6c5d71ead3a into 9923316413824b0e807ef015f987c70f(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:57:18,124 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:18,124 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a., storeName=d690627297fb3f579eb1d6c5d71ead3a/A, priority=13, startTime=1680670638043; duration=0sec 2023-04-05 04:57:18,124 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:57:18,125 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:57:18,126 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:57:18,126 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): d690627297fb3f579eb1d6c5d71ead3a/C is initiating minor compaction (all files) 2023-04-05 04:57:18,126 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d690627297fb3f579eb1d6c5d71ead3a/C in TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:18,126 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/fdcdce7652c84cd2ad63738fd66c27bf, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/4902aed2c06a4511864e5c1ed071ee78, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/764ba28c608f4718895e8cf8217945bd] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp, totalSize=34.9 K 2023-04-05 04:57:18,128 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting fdcdce7652c84cd2ad63738fd66c27bf, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1680670635099 2023-04-05 04:57:18,130 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 4902aed2c06a4511864e5c1ed071ee78, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1680670635795 2023-04-05 04:57:18,130 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 764ba28c608f4718895e8cf8217945bd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1680670636920 2023-04-05 04:57:18,150 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/3d1a803bb9da40d09b223da411e1e9db as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/3d1a803bb9da40d09b223da411e1e9db 2023-04-05 04:57:18,159 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d690627297fb3f579eb1d6c5d71ead3a/B of d690627297fb3f579eb1d6c5d71ead3a into 3d1a803bb9da40d09b223da411e1e9db(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:57:18,159 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:18,159 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a., storeName=d690627297fb3f579eb1d6c5d71ead3a/B, priority=13, startTime=1680670638045; duration=0sec 2023-04-05 04:57:18,159 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:57:18,161 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): d690627297fb3f579eb1d6c5d71ead3a#C#compaction#212 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:57:18,162 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/8258a49745c34790a345b242f5d591da 2023-04-05 04:57:18,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on d690627297fb3f579eb1d6c5d71ead3a 2023-04-05 04:57:18,190 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:18,199 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:18,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1680670698196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:18,199 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:18,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41420 deadline: 1680670698197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:18,201 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:18,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41454 deadline: 1680670698198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:18,202 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:18,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41430 deadline: 1680670698199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:18,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:18,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1680670698300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:18,302 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:18,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41420 deadline: 1680670698300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:18,305 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:18,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41454 deadline: 1680670698302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:18,305 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:18,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41430 deadline: 1680670698303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:18,504 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:18,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1680670698502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:18,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:18,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41420 deadline: 1680670698504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:18,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:18,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41430 deadline: 1680670698505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:18,507 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:18,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41454 deadline: 1680670698506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:18,591 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/526986ff88b042df82bc67adb04a27d9 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/526986ff88b042df82bc67adb04a27d9 2023-04-05 04:57:18,593 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/066c254ebe9f40e1972994af802ef689 2023-04-05 04:57:18,596 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d690627297fb3f579eb1d6c5d71ead3a/C of d690627297fb3f579eb1d6c5d71ead3a into 526986ff88b042df82bc67adb04a27d9(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:57:18,596 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:18,596 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a., storeName=d690627297fb3f579eb1d6c5d71ead3a/C, priority=13, startTime=1680670638045; duration=0sec 2023-04-05 04:57:18,596 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:57:18,806 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:18,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1680670698805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:18,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:18,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41420 deadline: 1680670698807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:18,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:18,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41430 deadline: 1680670698810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:18,811 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:18,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41454 deadline: 1680670698810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,005 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/ed96b7557619406b8fe50d34c9acd7d5 2023-04-05 04:57:19,008 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/8258a49745c34790a345b242f5d591da as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/8258a49745c34790a345b242f5d591da 2023-04-05 04:57:19,012 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/8258a49745c34790a345b242f5d591da, entries=150, sequenceid=144, filesize=11.8 K 2023-04-05 04:57:19,012 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/066c254ebe9f40e1972994af802ef689 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/066c254ebe9f40e1972994af802ef689 2023-04-05 04:57:19,016 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/066c254ebe9f40e1972994af802ef689, entries=150, sequenceid=144, filesize=11.8 K 2023-04-05 04:57:19,016 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/ed96b7557619406b8fe50d34c9acd7d5 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/ed96b7557619406b8fe50d34c9acd7d5 2023-04-05 04:57:19,019 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/ed96b7557619406b8fe50d34c9acd7d5, entries=150, sequenceid=144, filesize=11.8 K 2023-04-05 04:57:19,020 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for d690627297fb3f579eb1d6c5d71ead3a in 902ms, sequenceid=144, compaction requested=false 2023-04-05 04:57:19,020 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:19,020 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:19,020 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:19,021 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:19,021 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,021 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:19,021 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:19,023 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,023 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:19,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:19,024 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,024 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:19,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:19,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:19,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:19,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:19,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:19,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f31b03e[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:19,025 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:19,025 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,025 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,025 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:19,025 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:19,025 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,025 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:19,025 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:19,026 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:19,026 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,026 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:19,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:19,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:19,026 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:19,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:19,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:19,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:19,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:19,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:19,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:19,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:19,028 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:19,028 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:19,028 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:19,028 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:19,028 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:19,029 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,029 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:19,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:19,029 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:19,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:19,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:19,030 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:19,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:19,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:19,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:19,030 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:19,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:19,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,032 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:19,032 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:19,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:19,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:19,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:19,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:19,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:19,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:19,032 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:19,032 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:19,032 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:19,032 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,033 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:19,033 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:19,033 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,033 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,033 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,033 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,133 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:19,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:19,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:19,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:19,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:19,136 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:19,136 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:19,136 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:19,136 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:19,136 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,136 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:19,137 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,137 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:19,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:19,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:19,137 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,137 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:19,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:19,137 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:19,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:19,138 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:19,138 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:19,138 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:19,138 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:19,138 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:19,138 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:19,138 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:19,138 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:19,138 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:19,138 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:19,139 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d690627297fb3f579eb1d6c5d71ead3a 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 04:57:19,139 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=A 2023-04-05 04:57:19,139 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:19,139 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=B 2023-04-05 04:57:19,139 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:19,139 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=C 2023-04-05 04:57:19,139 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:19,149 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/fb3d74f18d3d4d43877076ba725e6186 2023-04-05 04:57:19,162 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/f3ba9e4bb54a47fe9f0f1580493ba681 2023-04-05 04:57:19,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on d690627297fb3f579eb1d6c5d71ead3a 2023-04-05 04:57:19,310 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:19,327 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:19,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680670639317, max timestamp=1680670639325Num uniques 859; 2023-04-05 04:57:19,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41430 deadline: 1680670699325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680670639317, max timestamp=1680670639325Num uniques 859; 2023-04-05 04:57:19,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 04:57:19,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680670639317, max timestamp=1680670639325Num uniques 859; 2023-04-05 04:57:19,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 04:57:19,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 04:57:19,344 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:19,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41420 deadline: 1680670699342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:19,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41454 deadline: 1680670699343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:19,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1680670699343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,430 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:19,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41430 deadline: 1680670699428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,447 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:19,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:19,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41420 deadline: 1680670699445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41454 deadline: 1680670699446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:19,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1680670699446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,574 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/52d9161460d64d6781300459898d67ed 2023-04-05 04:57:19,577 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/fb3d74f18d3d4d43877076ba725e6186 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/fb3d74f18d3d4d43877076ba725e6186 2023-04-05 04:57:19,581 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/fb3d74f18d3d4d43877076ba725e6186, entries=150, sequenceid=158, filesize=11.8 K 2023-04-05 04:57:19,581 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/f3ba9e4bb54a47fe9f0f1580493ba681 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/f3ba9e4bb54a47fe9f0f1580493ba681 2023-04-05 04:57:19,585 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/f3ba9e4bb54a47fe9f0f1580493ba681, entries=150, sequenceid=158, filesize=11.8 K 2023-04-05 04:57:19,586 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/52d9161460d64d6781300459898d67ed as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/52d9161460d64d6781300459898d67ed 2023-04-05 04:57:19,592 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/52d9161460d64d6781300459898d67ed, entries=150, sequenceid=158, filesize=11.8 K 2023-04-05 04:57:19,592 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for d690627297fb3f579eb1d6c5d71ead3a in 453ms, sequenceid=158, compaction requested=true 2023-04-05 04:57:19,592 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:19,592 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:19,592 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:19,592 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:19,593 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,593 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:19,593 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:19,593 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,594 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:19,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:19,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:19,594 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,594 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:19,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:19,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:19,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:19,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:19,595 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:19,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70a87a98[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:19,595 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,596 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,596 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:19,596 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:19,596 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,596 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:19,596 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:19,596 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,596 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:19,597 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:19,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:19,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:19,597 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:19,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:19,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:19,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:19,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:19,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:19,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:19,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:19,599 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:19,599 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:19,599 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:19,599 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:19,599 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:19,600 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,600 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:19,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,600 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:19,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:19,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:19,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:19,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:19,601 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:19,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:19,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:19,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,602 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:19,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:19,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,603 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:19,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:19,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:19,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:19,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:19,604 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:19,604 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:19,604 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,604 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:19,604 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:19,604 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:19,604 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:19,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:19,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:19,604 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,604 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,604 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,604 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on d690627297fb3f579eb1d6c5d71ead3a 2023-04-05 04:57:19,634 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d690627297fb3f579eb1d6c5d71ead3a 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-04-05 04:57:19,635 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=A 2023-04-05 04:57:19,635 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:19,635 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=B 2023-04-05 04:57:19,635 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:19,635 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=C 2023-04-05 04:57:19,635 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:19,651 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:19,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41420 deadline: 1680670699649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,651 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:19,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41454 deadline: 1680670699649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,651 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:19,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1680670699649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,653 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:19,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41430 deadline: 1680670699651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,704 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:19,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:19,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:19,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:19,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:19,707 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:19,707 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:19,707 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:19,707 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:19,709 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,709 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:19,710 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,710 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:19,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:19,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:19,710 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,710 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:19,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:19,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:19,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:19,711 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:19,711 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:19,711 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:19,711 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:19,711 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:19,711 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:19,711 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:19,711 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:19,711 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:19,711 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:19,711 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:19,711 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:19,711 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:19,711 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:19,711 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,711 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:19,712 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:19,712 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,712 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:19,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:19,713 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,713 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:19,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:19,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:19,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:19,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:19,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:19,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62fd5bcf[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:19,714 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:19,714 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,715 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,715 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:19,715 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:19,715 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:19,715 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,715 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:19,715 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:19,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:19,715 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:19,716 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:19,716 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:19,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:19,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:19,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:19,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:19,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:19,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:19,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:19,717 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:19,717 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:19,717 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:19,717 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:19,717 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:19,718 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,718 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:19,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:19,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,718 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:19,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:19,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:19,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:19,719 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:19,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:19,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:19,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,720 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:19,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:19,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,723 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:19,723 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:19,723 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:19,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:19,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:19,723 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:19,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:19,723 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:19,723 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:19,723 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:19,723 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:19,723 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,723 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,723 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,723 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:19,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:19,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:19,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41430 deadline: 1680670699754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,823 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:19,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:19,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:19,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:19,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:19,825 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:19,825 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:19,825 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:19,826 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:19,826 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,826 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:19,826 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,826 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:19,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:19,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:19,827 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,827 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:19,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:19,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:19,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:19,827 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:19,827 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:19,828 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:19,828 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:19,828 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:19,828 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:19,828 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:19,828 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:19,828 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:19,828 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:19,828 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:19,828 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:19,828 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:19,828 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:19,828 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,828 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:19,828 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:19,829 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,829 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:19,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:19,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:19,829 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,829 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:19,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:19,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79f86692[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:19,830 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:19,830 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,832 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,832 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:19,832 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:19,832 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:19,832 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,832 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:19,833 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,833 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:19,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:19,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:19,833 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:19,833 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:19,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:19,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:19,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:19,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:19,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:19,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:19,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:19,835 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:19,835 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:19,835 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:19,835 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:19,835 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:19,836 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,836 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,836 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:19,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:19,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:19,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,836 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:19,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:19,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:19,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:19,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:19,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:19,837 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:19,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:19,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,838 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:19,838 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:19,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:19,838 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:19,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:19,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:19,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:19,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:19,839 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:19,839 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:19,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:19,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:19,842 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,842 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:19,842 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:19,842 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,842 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,842 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,842 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,939 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:19,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:19,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:19,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:19,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:19,942 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:19,942 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:19,942 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:19,942 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:19,942 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,942 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:19,945 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:19,945 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:19,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:19,946 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,946 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:19,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:19,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:19,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:19,946 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:19,947 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:19,947 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:19,947 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:19,947 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:19,947 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:19,947 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:19,947 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:19,947 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:19,947 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:19,947 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:19,947 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:19,947 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:19,947 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:19,947 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,947 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:19,947 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:19,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41488 deadline: 1680670699946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,947 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:19,949 DEBUG [Thread-1612] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4150 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a., hostname=jenkins-hbase20.apache.org,38091,1680670569206, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 04:57:19,949 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,949 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:19,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:19,949 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,949 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:19,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:19,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:19,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:19,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:19,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:19,951 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:19,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bca89d3[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:19,951 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,951 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,951 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:19,951 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:19,951 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:19,951 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,951 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:19,952 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:19,952 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,952 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:19,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:19,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:19,952 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:19,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:19,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:19,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:19,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,953 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:19,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:19,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41454 deadline: 1680670699953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:19,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:19,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:19,954 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:19,955 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:19,955 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:19,955 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:19,955 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:19,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41420 deadline: 1680670699953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,955 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:19,956 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,956 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:19,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:19,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,956 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:19,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:19,956 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:19,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:19,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:19,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:19,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:19,957 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:19,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1680670699954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:19,957 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:19,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:19,958 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:19,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41430 deadline: 1680670699955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,959 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,959 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:19,959 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:19,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:19,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:19,959 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:19,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:19,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:19,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:19,959 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:19,959 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:19,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:19,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:19,959 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:19,959 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:19,959 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,960 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:19,960 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:19,960 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,044 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/87227a068d3d4b838c82cbba2437c6ec 2023-04-05 04:57:20,060 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:20,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:20,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:20,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:20,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:20,062 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:20,062 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:20,062 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:20,062 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:20,063 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,063 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:20,063 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,063 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:20,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:20,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:20,064 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,064 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:20,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:20,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:20,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:20,064 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:20,064 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:20,064 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:20,064 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:20,064 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:20,064 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:20,065 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:20,065 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:20,065 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:20,065 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:20,065 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:20,065 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:20,065 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:20,065 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:20,065 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,065 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:20,065 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:20,066 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,066 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:20,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:20,066 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:20,066 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:20,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:20,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:20,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:20,067 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:20,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48854088[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:20,067 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,068 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,068 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:20,068 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:20,068 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,068 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:20,068 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:20,069 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,069 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:20,069 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:20,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:20,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:20,069 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:20,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:20,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:20,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:20,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:20,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:20,071 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:20,071 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:20,071 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:20,071 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:20,071 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:20,072 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,072 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,072 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:20,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:20,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:20,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:20,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,073 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:20,073 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:20,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,075 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:20,075 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:20,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:20,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:20,075 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,075 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:20,075 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:20,075 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,075 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,075 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:20,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:20,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:20,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:20,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:20,075 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,076 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:20,076 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:20,076 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:20,176 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:20,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:20,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:20,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:20,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:20,178 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:20,179 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:20,179 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:20,179 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:20,179 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,179 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:20,187 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:20,187 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:20,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:20,187 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,187 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:20,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:20,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:20,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:20,188 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:20,188 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:20,188 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:20,188 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:20,188 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:20,188 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:20,188 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:20,188 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:20,188 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:20,188 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:20,188 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:20,188 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:20,188 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:20,188 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:20,188 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,188 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:20,188 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:20,191 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,191 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:20,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:20,192 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,192 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:20,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:20,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:20,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:20,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:20,193 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:20,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40ae4974[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:20,193 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,193 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,194 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:20,194 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:20,194 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:20,194 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,194 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:20,195 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:20,195 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,195 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:20,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:20,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:20,195 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:20,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:20,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:20,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:20,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:20,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:20,197 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:20,197 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:20,197 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:20,197 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:20,197 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:20,198 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,198 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,198 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:20,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:20,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:20,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:20,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:20,198 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:20,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:20,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,199 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:20,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,200 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:20,200 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,201 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:20,201 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:20,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:20,201 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:20,201 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:20,201 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:20,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:20,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:20,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:20,201 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,201 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,201 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:20,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:20,201 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:20,201 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:20,261 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:20,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41430 deadline: 1680670700260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,301 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:20,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:20,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:20,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:20,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:20,304 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:20,304 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:20,304 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:20,304 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:20,304 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,304 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:20,305 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,305 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:20,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:20,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:20,305 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,305 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:20,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:20,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:20,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:20,306 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:20,306 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:20,306 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:20,306 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:20,306 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:20,306 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:20,307 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:20,307 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:20,307 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:20,307 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:20,307 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:20,307 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:20,307 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:20,307 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:20,307 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,307 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:20,307 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:20,308 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,308 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:20,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:20,308 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,308 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:20,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:20,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:20,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:20,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:20,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a587b05[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:20,309 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:20,309 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,310 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,310 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:20,310 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:20,310 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:20,310 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,310 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:20,310 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:20,310 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,310 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:20,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:20,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:20,311 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:20,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:20,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:20,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:20,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:20,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:20,313 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:20,313 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:20,313 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:20,313 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:20,313 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:20,314 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,314 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,314 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:20,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:20,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:20,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:20,314 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:20,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:20,316 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:20,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,317 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,317 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:20,317 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:20,317 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:20,317 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,317 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:20,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:20,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:20,317 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:20,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:20,317 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:20,317 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,317 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:20,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:20,317 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:20,317 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:20,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:20,418 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:20,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:20,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:20,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:20,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:20,420 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:20,420 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:20,420 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:20,420 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:20,420 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,420 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:20,421 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:20,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:20,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:20,421 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:20,421 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,421 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:20,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:20,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:20,422 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:20,422 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:20,422 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:20,422 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:20,422 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:20,422 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:20,422 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:20,422 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:20,422 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:20,423 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:20,423 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:20,423 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:20,423 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:20,423 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:20,423 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,423 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:20,423 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:20,424 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,424 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:20,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:20,424 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,424 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:20,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:20,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:20,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:20,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:20,425 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:20,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55f3e18c[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:20,425 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,426 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,426 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,426 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:20,426 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:20,426 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:20,426 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:20,426 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,426 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:20,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:20,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:20,426 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:20,427 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:20,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:20,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:20,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:20,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:20,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:20,428 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:20,428 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:20,428 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:20,428 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:20,428 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:20,429 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,429 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,429 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:20,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:20,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:20,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:20,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:20,430 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:20,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:20,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,431 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:20,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,432 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:20,432 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:20,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:20,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:20,432 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:20,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:20,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:20,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:20,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:20,432 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:20,432 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:20,432 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:20,432 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:20,432 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:20,433 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,433 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,433 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,433 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,453 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/6b8f353e24594de498f671461c898f51 2023-04-05 04:57:20,457 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:20,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41454 deadline: 1680670700455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,459 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:20,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41420 deadline: 1680670700458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:20,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1680670700461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,533 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:20,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:20,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:20,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:20,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:20,535 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:20,535 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:20,535 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:20,535 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:20,535 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,535 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:20,536 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,536 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:20,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:20,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:20,536 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,536 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:20,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:20,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:20,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:20,537 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:20,537 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:20,537 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:20,537 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:20,537 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:20,537 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:20,537 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:20,537 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:20,537 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:20,538 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:20,538 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:20,538 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:20,538 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:20,538 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:20,538 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,538 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:20,538 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:20,539 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,539 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:20,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:20,539 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,539 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:20,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:20,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:20,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:20,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:20,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37106aba[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:20,540 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:20,540 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,540 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,541 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:20,541 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:20,541 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:20,541 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,541 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:20,541 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,541 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:20,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:20,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:20,541 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:20,541 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:20,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:20,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:20,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:20,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:20,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:20,543 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:20,543 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:20,543 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:20,543 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:20,543 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:20,545 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,545 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:20,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:20,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,545 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:20,545 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:20,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:20,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:20,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:20,546 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:20,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,547 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:20,547 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:20,547 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:20,547 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:20,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:20,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:20,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:20,547 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:20,548 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:20,548 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:20,548 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:20,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:20,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:20,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:20,548 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,548 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,548 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,548 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,648 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:20,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:20,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:20,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:20,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:20,650 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:20,650 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:20,650 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:20,651 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:20,651 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,651 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:20,651 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:20,651 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:20,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:20,652 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,652 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:20,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:20,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:20,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:20,652 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:20,653 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:20,653 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:20,653 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:20,653 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:20,653 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:20,653 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:20,653 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:20,653 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:20,653 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:20,653 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:20,653 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:20,653 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:20,653 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:20,653 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,653 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:20,653 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:20,654 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,654 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:20,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:20,655 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,655 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:20,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:20,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:20,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:20,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:20,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c760c5[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:20,656 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:20,656 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,657 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,657 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:20,657 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,657 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:20,657 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:20,657 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:20,657 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,657 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:20,657 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:20,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:20,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:20,657 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:20,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:20,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:20,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:20,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:20,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:20,659 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:20,659 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:20,659 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:20,660 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:20,660 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:20,660 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,660 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,660 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:20,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:20,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:20,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:20,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:20,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:20,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,661 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:20,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:20,662 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,664 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:20,664 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:20,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:20,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:20,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:20,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:20,664 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:20,664 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,664 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:20,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:20,664 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:20,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:20,664 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:20,664 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:20,664 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,664 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,664 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,664 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,764 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:20,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:20,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:20,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:20,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:20,767 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:20,767 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:20,767 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:20,767 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:20,768 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:20,768 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,768 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:20,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41430 deadline: 1680670700766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,768 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,769 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:20,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:20,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:20,769 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,769 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:20,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:20,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:20,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:20,770 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:20,770 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:20,770 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:20,770 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:20,770 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:20,770 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:20,770 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:20,770 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:20,770 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:20,770 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:20,770 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:20,770 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:20,771 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:20,771 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:20,771 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,771 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:20,771 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:20,772 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,772 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:20,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:20,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:20,772 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,772 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:20,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:20,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:20,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:20,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cfeb32b[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:20,773 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:20,773 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,774 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,774 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:20,774 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:20,774 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,774 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:20,774 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:20,775 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:20,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:20,775 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:20,775 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:20,775 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:20,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:20,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:20,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:20,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:20,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:20,777 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:20,777 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:20,777 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:20,777 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:20,777 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:20,778 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,778 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:20,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,779 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:20,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:20,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:20,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:20,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:20,779 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:20,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,781 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:20,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:20,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,791 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,791 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:20,792 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:20,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:20,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:20,792 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:20,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:20,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:20,792 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:20,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:20,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:20,792 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:20,792 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:20,793 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:20,793 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,793 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,793 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:20,793 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,868 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/a19bafbf862e4c5baf27c8ab51032768 2023-04-05 04:57:20,871 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/87227a068d3d4b838c82cbba2437c6ec as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/87227a068d3d4b838c82cbba2437c6ec 2023-04-05 04:57:20,874 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/87227a068d3d4b838c82cbba2437c6ec, entries=200, sequenceid=194, filesize=14.1 K 2023-04-05 04:57:20,875 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/6b8f353e24594de498f671461c898f51 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/6b8f353e24594de498f671461c898f51 2023-04-05 04:57:20,893 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:20,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:20,893 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/6b8f353e24594de498f671461c898f51, entries=150, sequenceid=194, filesize=11.8 K 2023-04-05 04:57:20,894 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/a19bafbf862e4c5baf27c8ab51032768 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/a19bafbf862e4c5baf27c8ab51032768 2023-04-05 04:57:20,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:20,900 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/a19bafbf862e4c5baf27c8ab51032768, entries=150, sequenceid=194, filesize=11.8 K 2023-04-05 04:57:20,901 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=40.25 KB/41220 for d690627297fb3f579eb1d6c5d71ead3a in 1266ms, sequenceid=194, compaction requested=true 2023-04-05 04:57:20,901 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:20,901 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:57:20,901 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:57:20,901 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:57:20,901 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:57:20,901 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:57:20,902 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50590 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:57:20,902 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:57:20,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:20,902 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): d690627297fb3f579eb1d6c5d71ead3a/B is initiating minor compaction (all files) 2023-04-05 04:57:20,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:20,902 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): d690627297fb3f579eb1d6c5d71ead3a/A is initiating minor compaction (all files) 2023-04-05 04:57:20,902 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d690627297fb3f579eb1d6c5d71ead3a/B in TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:20,902 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d690627297fb3f579eb1d6c5d71ead3a/A in TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:20,902 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:20,902 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/3d1a803bb9da40d09b223da411e1e9db, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/066c254ebe9f40e1972994af802ef689, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/f3ba9e4bb54a47fe9f0f1580493ba681, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/6b8f353e24594de498f671461c898f51] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp, totalSize=47.1 K 2023-04-05 04:57:20,902 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:20,902 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/9923316413824b0e807ef015f987c70f, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/8258a49745c34790a345b242f5d591da, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/fb3d74f18d3d4d43877076ba725e6186, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/87227a068d3d4b838c82cbba2437c6ec] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp, totalSize=49.4 K 2023-04-05 04:57:20,902 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:20,902 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:20,902 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 3d1a803bb9da40d09b223da411e1e9db, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1680670636920 2023-04-05 04:57:20,903 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,903 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:20,903 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 066c254ebe9f40e1972994af802ef689, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1680670637577 2023-04-05 04:57:20,903 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 9923316413824b0e807ef015f987c70f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1680670636920 2023-04-05 04:57:20,903 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting f3ba9e4bb54a47fe9f0f1580493ba681, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1680670638194 2023-04-05 04:57:20,903 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:20,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:20,904 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 6b8f353e24594de498f671461c898f51, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1680670639342 2023-04-05 04:57:20,904 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,904 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 8258a49745c34790a345b242f5d591da, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1680670637577 2023-04-05 04:57:20,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:20,904 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:20,904 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:20,904 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting fb3d74f18d3d4d43877076ba725e6186, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1680670638194 2023-04-05 04:57:20,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:20,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:20,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:20,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:20,905 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:20,905 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 87227a068d3d4b838c82cbba2437c6ec, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1680670639339 2023-04-05 04:57:20,913 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:20,913 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:20,913 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:20,913 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:20,913 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:20,913 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:20,913 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:20,913 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:20,913 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:20,914 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d690627297fb3f579eb1d6c5d71ead3a 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-05 04:57:20,914 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=A 2023-04-05 04:57:20,915 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:20,915 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=B 2023-04-05 04:57:20,915 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:20,915 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=C 2023-04-05 04:57:20,915 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:20,923 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): d690627297fb3f579eb1d6c5d71ead3a#B#compaction#221 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:57:20,924 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): d690627297fb3f579eb1d6c5d71ead3a#A#compaction#222 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:57:20,961 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/77adc92b05294903a0f6e1ffc02de278 2023-04-05 04:57:20,961 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/a98067121af740638dfbec3d0caf259c as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/a98067121af740638dfbec3d0caf259c 2023-04-05 04:57:20,964 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/a80eea887a94448996db0e8ab4cbf41c as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/a80eea887a94448996db0e8ab4cbf41c 2023-04-05 04:57:20,969 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d690627297fb3f579eb1d6c5d71ead3a/A of d690627297fb3f579eb1d6c5d71ead3a into a98067121af740638dfbec3d0caf259c(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:57:20,969 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:20,969 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a., storeName=d690627297fb3f579eb1d6c5d71ead3a/A, priority=12, startTime=1680670640901; duration=0sec 2023-04-05 04:57:20,969 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:57:20,969 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:57:20,970 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:57:20,970 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): d690627297fb3f579eb1d6c5d71ead3a/C is initiating minor compaction (all files) 2023-04-05 04:57:20,970 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d690627297fb3f579eb1d6c5d71ead3a/C in TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:20,970 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/526986ff88b042df82bc67adb04a27d9, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/ed96b7557619406b8fe50d34c9acd7d5, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/52d9161460d64d6781300459898d67ed, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/a19bafbf862e4c5baf27c8ab51032768] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp, totalSize=47.1 K 2023-04-05 04:57:20,971 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 526986ff88b042df82bc67adb04a27d9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1680670636920 2023-04-05 04:57:20,971 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting ed96b7557619406b8fe50d34c9acd7d5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1680670637577 2023-04-05 04:57:20,972 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 52d9161460d64d6781300459898d67ed, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1680670638194 2023-04-05 04:57:20,973 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting a19bafbf862e4c5baf27c8ab51032768, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1680670639342 2023-04-05 04:57:20,996 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): d690627297fb3f579eb1d6c5d71ead3a#C#compaction#225 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:57:20,996 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d690627297fb3f579eb1d6c5d71ead3a/B of d690627297fb3f579eb1d6c5d71ead3a into a80eea887a94448996db0e8ab4cbf41c(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:57:20,996 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:20,996 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a., storeName=d690627297fb3f579eb1d6c5d71ead3a/B, priority=12, startTime=1680670640901; duration=0sec 2023-04-05 04:57:20,996 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:57:21,046 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/f2e3a4aa8b7b46f78af865ef6eb4fb8f as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/f2e3a4aa8b7b46f78af865ef6eb4fb8f 2023-04-05 04:57:21,050 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/e34048afa1c14c8e8104113a553c894e 2023-04-05 04:57:21,051 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d690627297fb3f579eb1d6c5d71ead3a/C of d690627297fb3f579eb1d6c5d71ead3a into f2e3a4aa8b7b46f78af865ef6eb4fb8f(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:57:21,051 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:21,051 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a., storeName=d690627297fb3f579eb1d6c5d71ead3a/C, priority=12, startTime=1680670640901; duration=0sec 2023-04-05 04:57:21,051 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:57:21,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on d690627297fb3f579eb1d6c5d71ead3a 2023-04-05 04:57:21,461 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:21,470 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/7c17be52b9b441bfb50c9b2b023c76df 2023-04-05 04:57:21,473 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/77adc92b05294903a0f6e1ffc02de278 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/77adc92b05294903a0f6e1ffc02de278 2023-04-05 04:57:21,476 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/77adc92b05294903a0f6e1ffc02de278, entries=150, sequenceid=203, filesize=11.8 K 2023-04-05 04:57:21,477 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/e34048afa1c14c8e8104113a553c894e as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/e34048afa1c14c8e8104113a553c894e 2023-04-05 04:57:21,480 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/e34048afa1c14c8e8104113a553c894e, entries=150, sequenceid=203, filesize=11.8 K 2023-04-05 04:57:21,481 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/7c17be52b9b441bfb50c9b2b023c76df as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/7c17be52b9b441bfb50c9b2b023c76df 2023-04-05 04:57:21,484 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/7c17be52b9b441bfb50c9b2b023c76df, entries=150, sequenceid=203, filesize=11.8 K 2023-04-05 04:57:21,485 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=120.76 KB/123660 for d690627297fb3f579eb1d6c5d71ead3a in 571ms, sequenceid=203, compaction requested=false 2023-04-05 04:57:21,486 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:21,486 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670641477, max timestamp=1680670641484Num uniques 859; 2023-04-05 04:57:21,486 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670641477, max timestamp=1680670641484Num uniques 859; 2023-04-05 04:57:21,486 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:57:21,486 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:21,486 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670641477, max timestamp=1680670641484Num uniques 859; 2023-04-05 04:57:21,486 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:57:21,486 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:57:21,486 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:21,486 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:21,486 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,486 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:21,486 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:21,487 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,487 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:21,487 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,487 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:21,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:21,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:21,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:21,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:21,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76cbdab9[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:21,489 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:21,489 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,489 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,489 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:21,489 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:21,489 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:21,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on d690627297fb3f579eb1d6c5d71ead3a 2023-04-05 04:57:21,490 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,490 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:21,490 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d690627297fb3f579eb1d6c5d71ead3a 3/3 column families, dataSize=53.67 KB heapSize=145.92 KB 2023-04-05 04:57:21,490 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:21,490 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:21,490 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,490 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:21,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:21,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:21,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:21,491 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=A 2023-04-05 04:57:21,491 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:21,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:21,491 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=B 2023-04-05 04:57:21,491 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:21,491 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=C 2023-04-05 04:57:21,491 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:21,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:21,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:21,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:21,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:21,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:21,493 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:21,493 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:21,493 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:21,493 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:21,493 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:21,494 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:21,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:21,494 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,495 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:21,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,495 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:21,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:21,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:21,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:21,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:21,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:21,496 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,499 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,499 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:21,499 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:21,499 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:21,499 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:21,499 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:21,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:21,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:21,499 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:21,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:21,499 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:21,499 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,499 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:21,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:21,499 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,499 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:21,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:21,512 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:21,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41420 deadline: 1680670701509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,513 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670641492, max timestamp=1680670641510Num uniques 859; 2023-04-05 04:57:21,513 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670641492, max timestamp=1680670641510Num uniques 859; 2023-04-05 04:57:21,513 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:57:21,513 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670641492, max timestamp=1680670641510Num uniques 859; 2023-04-05 04:57:21,513 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:57:21,513 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:57:21,519 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/d2ae4f3caae34c5c9512206d89efd486 2023-04-05 04:57:21,539 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:21,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41454 deadline: 1680670701536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,540 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:21,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1680670701537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,599 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:21,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:21,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:21,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:21,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:21,602 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:21,602 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:21,602 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:21,602 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:21,603 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,603 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:21,603 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:21,603 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:21,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:21,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:21,604 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,604 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:21,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:21,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:21,604 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:21,605 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:21,605 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:21,605 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:21,605 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:21,605 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:21,605 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:21,605 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:21,605 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:21,605 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:21,605 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:21,605 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:21,605 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:21,605 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:21,605 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,605 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:21,605 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:21,606 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,606 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:21,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:21,606 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,607 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:21,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:21,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:21,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:21,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:21,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:21,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@423a2b60[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:21,608 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:21,608 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,608 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,608 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:21,609 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,609 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:21,609 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:21,609 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:21,609 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,609 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:21,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:21,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:21,609 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:21,610 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:21,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:21,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:21,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:21,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:21,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:21,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:21,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:21,611 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:21,611 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:21,612 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:21,612 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:21,612 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:21,612 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,612 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:21,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:21,613 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:21,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:21,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:21,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:21,613 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:21,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:21,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:21,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,614 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:21,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:21,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,616 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:21,616 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:21,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41420 deadline: 1680670701614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,616 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:21,616 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:21,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:21,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:21,616 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:21,616 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:21,616 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:21,616 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:21,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:21,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:21,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:21,616 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,617 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,617 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,617 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:21,617 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:21,643 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:21,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41454 deadline: 1680670701640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,644 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:21,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1680670701641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,717 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:21,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:21,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:21,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:21,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:21,720 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:21,720 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:21,720 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:21,720 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:21,720 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,720 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:21,721 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:21,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:21,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:21,721 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:21,721 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,721 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:21,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:21,722 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:21,722 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:21,722 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:21,722 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:21,722 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:21,722 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:21,722 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:21,722 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:21,722 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:21,722 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:21,722 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:21,722 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:21,722 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:21,722 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:21,722 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,722 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:21,722 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:21,723 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:21,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:21,723 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:21,724 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,724 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:21,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:21,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:21,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:21,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:21,725 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:21,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cc5baaa[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:21,725 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,725 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,725 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:21,725 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:21,725 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,725 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:21,725 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:21,726 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:21,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:21,726 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:21,726 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:21,726 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:21,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:21,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:21,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:21,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:21,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:21,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:21,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:21,728 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:21,728 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:21,728 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:21,728 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:21,728 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:21,729 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:21,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:21,729 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,729 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:21,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:21,729 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:21,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:21,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:21,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:21,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:21,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,730 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:21,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:21,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,732 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:21,732 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:21,732 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:21,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:21,732 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:21,732 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:21,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:21,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:21,732 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:21,732 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:21,732 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:21,732 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,732 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:21,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:21,732 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,733 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,771 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:21,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41430 deadline: 1680670701770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,820 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:21,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41420 deadline: 1680670701818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,832 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:21,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:21,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:21,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:21,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:21,835 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:21,835 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:21,835 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:21,835 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:21,835 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,835 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:21,836 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,836 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:21,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:21,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:21,836 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,836 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:21,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:21,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:21,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:21,837 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:21,837 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:21,837 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:21,837 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:21,837 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:21,837 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:21,838 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:21,838 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:21,838 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:21,838 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:21,838 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:21,838 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:21,838 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:21,838 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:21,838 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,838 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:21,838 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:21,839 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,839 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:21,839 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,839 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:21,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:21,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:21,840 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:21,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35eb29f2[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:21,840 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,841 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,841 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:21,841 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:21,841 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:21,841 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,841 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:21,841 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:21,841 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:21,842 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:21,842 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:21,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:21,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:21,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:21,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:21,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:21,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:21,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:21,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:21,844 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:21,844 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:21,844 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:21,844 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:21,844 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:21,845 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,845 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:21,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:21,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,845 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:21,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:21,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:21,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:21,845 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,846 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:21,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:21,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,848 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:21,848 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1680670701845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,848 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:21,848 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:21,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:21,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:21,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:21,848 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:21,848 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:21,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:21,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:21,848 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:21,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:21,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:21,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41454 deadline: 1680670701845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,848 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:21,848 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:21,848 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:21,848 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,848 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,848 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,848 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,929 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/c47359947b914dd89fc762524a8ab105 2023-04-05 04:57:21,942 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/2afefd0f86134e25996c9823c5b75325 2023-04-05 04:57:21,948 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/d2ae4f3caae34c5c9512206d89efd486 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/d2ae4f3caae34c5c9512206d89efd486 2023-04-05 04:57:21,948 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:21,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:21,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:21,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:21,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:21,953 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:21,953 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:21,953 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:21,953 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:21,953 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,953 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:21,954 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/d2ae4f3caae34c5c9512206d89efd486, entries=200, sequenceid=232, filesize=14.1 K 2023-04-05 04:57:21,955 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/c47359947b914dd89fc762524a8ab105 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/c47359947b914dd89fc762524a8ab105 2023-04-05 04:57:21,955 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,955 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:21,955 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:21,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:21,955 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:21,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:21,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:21,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:21,956 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:21,957 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:21,957 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:21,957 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:21,957 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:21,957 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:21,957 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:21,957 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:21,957 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:21,957 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:21,957 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:21,957 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:21,957 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:21,957 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:21,957 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,957 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:21,957 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:21,958 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,958 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:21,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:21,958 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/c47359947b914dd89fc762524a8ab105, entries=150, sequenceid=232, filesize=11.8 K 2023-04-05 04:57:21,959 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,959 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:21,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:21,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:21,959 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/2afefd0f86134e25996c9823c5b75325 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/2afefd0f86134e25996c9823c5b75325 2023-04-05 04:57:21,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:21,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:21,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:21,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ddd33e4[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:21,960 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:21,960 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,961 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,961 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:21,961 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:21,961 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:21,961 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,961 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:21,962 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:21,962 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:21,962 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:21,962 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/2afefd0f86134e25996c9823c5b75325, entries=150, sequenceid=232, filesize=11.8 K 2023-04-05 04:57:21,963 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:21,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:21,963 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~165.80 KB/169776, currentSize=125.23 KB/128240 for d690627297fb3f579eb1d6c5d71ead3a in 473ms, sequenceid=232, compaction requested=true 2023-04-05 04:57:21,963 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:21,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:21,963 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:57:21,963 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:57:21,963 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:57:21,963 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:57:21,963 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:57:21,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:21,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:21,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,964 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38840 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:57:21,964 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): d690627297fb3f579eb1d6c5d71ead3a/A is initiating minor compaction (all files) 2023-04-05 04:57:21,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:21,964 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d690627297fb3f579eb1d6c5d71ead3a/A in TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:21,964 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/a98067121af740638dfbec3d0caf259c, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/77adc92b05294903a0f6e1ffc02de278, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/d2ae4f3caae34c5c9512206d89efd486] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp, totalSize=37.9 K 2023-04-05 04:57:21,965 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:57:21,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:21,965 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): d690627297fb3f579eb1d6c5d71ead3a/B is initiating minor compaction (all files) 2023-04-05 04:57:21,965 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d690627297fb3f579eb1d6c5d71ead3a/B in TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:21,965 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/a80eea887a94448996db0e8ab4cbf41c, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/e34048afa1c14c8e8104113a553c894e, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/c47359947b914dd89fc762524a8ab105] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp, totalSize=35.6 K 2023-04-05 04:57:21,965 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting a98067121af740638dfbec3d0caf259c, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1680670639342 2023-04-05 04:57:21,965 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting a80eea887a94448996db0e8ab4cbf41c, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1680670639342 2023-04-05 04:57:21,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,966 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 77adc92b05294903a0f6e1ffc02de278, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680670639641 2023-04-05 04:57:21,966 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting e34048afa1c14c8e8104113a553c894e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680670639641 2023-04-05 04:57:21,966 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting d2ae4f3caae34c5c9512206d89efd486, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1680670641481 2023-04-05 04:57:21,966 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting c47359947b914dd89fc762524a8ab105, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1680670641481 2023-04-05 04:57:21,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:21,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:21,966 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:21,966 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:21,966 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:21,966 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:21,966 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:21,968 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,968 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:21,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,968 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:21,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:21,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:21,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:21,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:21,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:21,969 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:21,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:21,970 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:21,973 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): d690627297fb3f579eb1d6c5d71ead3a#B#compaction#230 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:57:21,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:21,974 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): d690627297fb3f579eb1d6c5d71ead3a#A#compaction#231 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:57:21,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,975 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:21,975 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:21,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:21,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:21,975 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:21,976 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:21,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:21,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:21,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:21,976 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:21,976 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:21,976 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:21,976 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,976 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:21,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:21,976 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:21,976 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:21,976 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:21,993 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/adec1d8788274ffb908c2021ec4fb471 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/adec1d8788274ffb908c2021ec4fb471 2023-04-05 04:57:21,999 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d690627297fb3f579eb1d6c5d71ead3a/A of d690627297fb3f579eb1d6c5d71ead3a into adec1d8788274ffb908c2021ec4fb471(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:57:21,999 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:21,999 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a., storeName=d690627297fb3f579eb1d6c5d71ead3a/A, priority=13, startTime=1680670641963; duration=0sec 2023-04-05 04:57:21,999 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:57:21,999 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:57:22,000 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:57:22,000 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): d690627297fb3f579eb1d6c5d71ead3a/C is initiating minor compaction (all files) 2023-04-05 04:57:22,001 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d690627297fb3f579eb1d6c5d71ead3a/C in TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:22,001 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/f2e3a4aa8b7b46f78af865ef6eb4fb8f, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/7c17be52b9b441bfb50c9b2b023c76df, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/2afefd0f86134e25996c9823c5b75325] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp, totalSize=35.6 K 2023-04-05 04:57:22,003 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting f2e3a4aa8b7b46f78af865ef6eb4fb8f, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1680670639342 2023-04-05 04:57:22,003 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 7c17be52b9b441bfb50c9b2b023c76df, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680670639641 2023-04-05 04:57:22,004 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 2afefd0f86134e25996c9823c5b75325, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1680670641481 2023-04-05 04:57:22,011 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): d690627297fb3f579eb1d6c5d71ead3a#C#compaction#232 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:57:22,076 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:22,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:22,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:22,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:22,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:22,079 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:22,079 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:22,079 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:22,079 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:22,079 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:22,079 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:22,080 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:22,080 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:22,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:22,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:22,080 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:22,080 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:22,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:22,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:22,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:22,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:22,081 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:22,081 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:22,081 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:22,081 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:22,081 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:22,081 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:22,081 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:22,081 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:22,081 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:22,082 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:22,082 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d690627297fb3f579eb1d6c5d71ead3a 3/3 column families, dataSize=125.23 KB heapSize=355.30 KB 2023-04-05 04:57:22,082 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=A 2023-04-05 04:57:22,082 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:22,082 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=B 2023-04-05 04:57:22,082 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:22,082 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=C 2023-04-05 04:57:22,082 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:22,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on d690627297fb3f579eb1d6c5d71ead3a 2023-04-05 04:57:22,122 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:22,142 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:22,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41420 deadline: 1680670702141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:22,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:22,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41454 deadline: 1680670702149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:22,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:22,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1680670702150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:22,244 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:22,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41420 deadline: 1680670702243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:22,394 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/f3a0d93bfaa8444eae4a6d4376251b62 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/f3a0d93bfaa8444eae4a6d4376251b62 2023-04-05 04:57:22,398 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d690627297fb3f579eb1d6c5d71ead3a/B of d690627297fb3f579eb1d6c5d71ead3a into f3a0d93bfaa8444eae4a6d4376251b62(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:57:22,398 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:22,398 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a., storeName=d690627297fb3f579eb1d6c5d71ead3a/B, priority=13, startTime=1680670641963; duration=0sec 2023-04-05 04:57:22,398 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:57:22,427 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/42ab4f5e223d4d1ca3ef2f12b4ce83dc as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/42ab4f5e223d4d1ca3ef2f12b4ce83dc 2023-04-05 04:57:22,431 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d690627297fb3f579eb1d6c5d71ead3a/C of d690627297fb3f579eb1d6c5d71ead3a into 42ab4f5e223d4d1ca3ef2f12b4ce83dc(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:57:22,431 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:22,431 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a., storeName=d690627297fb3f579eb1d6c5d71ead3a/C, priority=13, startTime=1680670641963; duration=0sec 2023-04-05 04:57:22,431 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:57:22,448 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:22,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41420 deadline: 1680670702446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:22,492 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/b30329d40e2349a48fe9dffd5a0a963d 2023-04-05 04:57:22,500 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/d1af6161321740ec9f5e55609266fc36 2023-04-05 04:57:22,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:22,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1680670702654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:22,656 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:22,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41454 deadline: 1680670702655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:22,749 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:22,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41420 deadline: 1680670702749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:22,911 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/8d6fb8800aa24e4996d4b52ff0269775 2023-04-05 04:57:22,914 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/b30329d40e2349a48fe9dffd5a0a963d as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/b30329d40e2349a48fe9dffd5a0a963d 2023-04-05 04:57:22,917 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/b30329d40e2349a48fe9dffd5a0a963d, entries=150, sequenceid=268, filesize=11.9 K 2023-04-05 04:57:22,918 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/d1af6161321740ec9f5e55609266fc36 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/d1af6161321740ec9f5e55609266fc36 2023-04-05 04:57:22,921 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/d1af6161321740ec9f5e55609266fc36, entries=150, sequenceid=268, filesize=11.9 K 2023-04-05 04:57:22,921 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/8d6fb8800aa24e4996d4b52ff0269775 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/8d6fb8800aa24e4996d4b52ff0269775 2023-04-05 04:57:22,925 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/8d6fb8800aa24e4996d4b52ff0269775, entries=150, sequenceid=268, filesize=11.9 K 2023-04-05 04:57:22,925 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~125.23 KB/128240, heapSize ~355.25 KB/363776, currentSize=53.67 KB/54960 for d690627297fb3f579eb1d6c5d71ead3a in 843ms, sequenceid=268, compaction requested=false 2023-04-05 04:57:22,925 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:22,926 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:22,926 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:22,926 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:22,926 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:22,926 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:22,926 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:22,927 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:22,927 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:22,927 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:22,927 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:22,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:22,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:22,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:22,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@eaf6a14[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:22,928 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:22,928 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:22,929 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:22,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:22,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:22,929 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:22,929 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:22,929 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:22,929 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:22,929 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:22,930 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:22,930 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:22,930 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:22,930 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:22,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:22,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:22,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:22,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:22,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:22,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:22,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:22,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:22,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:22,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:22,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:22,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:22,932 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:22,933 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:22,933 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:22,933 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:22,933 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:22,933 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:22,933 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:22,933 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:22,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:22,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:22,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:22,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:22,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:22,934 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:22,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:22,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:22,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:22,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:22,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:22,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:22,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:22,935 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:22,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:22,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:22,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:22,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:22,936 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:22,936 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:22,936 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:22,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:22,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:22,936 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:22,936 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:22,936 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:22,936 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:22,936 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:22,936 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:22,936 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:22,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:22,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:22,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:22,937 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:22,937 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:22,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:22,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:23,037 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:23,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:23,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:23,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:23,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:23,040 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:23,040 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:23,040 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:23,040 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:23,040 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:23,040 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:23,041 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:23,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:23,041 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:23,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:23,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:23,041 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:23,041 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:23,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:23,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:23,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:23,042 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:23,042 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:23,042 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:23,042 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:23,042 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:23,042 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:23,042 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:23,042 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:23,042 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:23,042 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:23,043 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d690627297fb3f579eb1d6c5d71ead3a 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 04:57:23,043 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=A 2023-04-05 04:57:23,043 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:23,043 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=B 2023-04-05 04:57:23,043 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:23,043 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=C 2023-04-05 04:57:23,043 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:23,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on d690627297fb3f579eb1d6c5d71ead3a 2023-04-05 04:57:23,255 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:23,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670643289, max timestamp=1680670643297Num uniques 859; 2023-04-05 04:57:23,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:57:23,299 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670643289, max timestamp=1680670643297Num uniques 859; 2023-04-05 04:57:23,299 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:57:23,299 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670643289, max timestamp=1680670643297Num uniques 859; 2023-04-05 04:57:23,299 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:57:23,362 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:23,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41420 deadline: 1680670703359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:23,452 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=281 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/9b12e5385102444a978e2d586bc00dca 2023-04-05 04:57:23,463 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=281 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/76fad2de8c5b43a8a6050baa5ce78411 2023-04-05 04:57:23,466 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:23,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41420 deadline: 1680670703463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:23,663 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:23,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1680670703661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:23,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:23,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41454 deadline: 1680670703664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:23,670 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:23,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41420 deadline: 1680670703667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:23,789 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:23,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41430 deadline: 1680670703784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:23,794 DEBUG [Thread-1614] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4139 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a., hostname=jenkins-hbase20.apache.org,38091,1680670569206, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 04:57:23,888 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=281 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/f5c062b08845408cab8418b68105784c 2023-04-05 04:57:23,902 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/9b12e5385102444a978e2d586bc00dca as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/9b12e5385102444a978e2d586bc00dca 2023-04-05 04:57:23,908 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/9b12e5385102444a978e2d586bc00dca, entries=150, sequenceid=281, filesize=11.9 K 2023-04-05 04:57:23,909 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/76fad2de8c5b43a8a6050baa5ce78411 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/76fad2de8c5b43a8a6050baa5ce78411 2023-04-05 04:57:23,935 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/76fad2de8c5b43a8a6050baa5ce78411, entries=150, sequenceid=281, filesize=11.9 K 2023-04-05 04:57:23,936 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/f5c062b08845408cab8418b68105784c as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/f5c062b08845408cab8418b68105784c 2023-04-05 04:57:23,947 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/f5c062b08845408cab8418b68105784c, entries=150, sequenceid=281, filesize=11.9 K 2023-04-05 04:57:23,948 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for d690627297fb3f579eb1d6c5d71ead3a in 906ms, sequenceid=281, compaction requested=true 2023-04-05 04:57:23,948 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:23,948 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:23,948 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:23,948 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:23,948 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:23,950 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:23,950 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:23,951 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:23,952 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:23,952 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:23,952 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:23,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:23,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:23,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:23,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:23,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:23,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:23,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:23,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:23,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:23,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f6e8181[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:23,954 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:23,954 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:23,955 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:23,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:23,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:23,955 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:23,955 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:23,955 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:23,955 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:23,955 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:23,956 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:23,956 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:23,956 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:23,962 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:23,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:23,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:23,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:23,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:23,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:23,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:23,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:23,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:23,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:23,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:23,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:23,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:23,965 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:23,965 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:23,965 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:23,965 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:23,965 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:23,966 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:23,966 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:23,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:23,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:23,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:23,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:23,967 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:23,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:23,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:23,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:23,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:23,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:23,969 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:23,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:23,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:23,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:23,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:23,970 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:23,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:23,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:23,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:23,975 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:23,975 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:23,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:23,975 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:23,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:23,975 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:23,975 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:23,975 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:23,975 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:23,975 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:23,975 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:23,975 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:23,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:23,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:23,976 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:23,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:23,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:23,976 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:23,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on d690627297fb3f579eb1d6c5d71ead3a 2023-04-05 04:57:23,979 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d690627297fb3f579eb1d6c5d71ead3a 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-05 04:57:23,980 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=A 2023-04-05 04:57:23,980 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:23,980 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=B 2023-04-05 04:57:23,980 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:23,980 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=C 2023-04-05 04:57:23,980 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:24,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:24,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41488 deadline: 1680670704006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:24,011 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:24,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41420 deadline: 1680670704007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:24,030 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/be5f13bb192c4555b0627c2187c25102 2023-04-05 04:57:24,076 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/a57987ebd5384b15a1a327ff0dc87907 2023-04-05 04:57:24,076 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:24,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:24,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:24,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:24,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:24,080 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:24,080 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:24,080 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:24,080 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:24,080 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:24,080 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:24,082 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:24,082 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:24,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:24,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:24,082 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:24,082 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:24,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:24,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:24,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:24,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:24,083 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:24,092 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:24,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:24,092 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:24,092 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:24,092 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:24,094 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:24,094 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:24,094 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:24,095 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:24,095 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:24,095 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:24,095 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:24,095 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:24,095 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:24,095 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:24,095 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:24,098 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:24,098 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:24,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:24,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:24,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:24,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:24,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:24,099 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:24,099 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:24,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:24,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:24,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:24,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:24,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33269727[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:24,101 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:24,101 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:24,109 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:24,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:24,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:24,109 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:24,109 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:24,109 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:24,110 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:24,110 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:24,110 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:24,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:24,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41488 deadline: 1680670704111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:24,117 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:24,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41420 deadline: 1680670704112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:24,118 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:24,118 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:24,118 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:24,118 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:24,118 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:24,118 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:24,118 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:24,118 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:24,119 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:24,119 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:24,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:24,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:24,119 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:24,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:24,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:24,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:24,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:24,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:24,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:24,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:24,120 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:24,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:24,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:24,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:24,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:24,122 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:24,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:24,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:24,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:24,133 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:24,133 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:24,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:24,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:24,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:24,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:24,133 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:24,133 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:24,133 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:24,133 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:24,133 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:24,133 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:24,133 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:24,133 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:24,133 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:24,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:24,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:24,133 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:24,170 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/69c6760c7d2a48dbab37dd28ddb3c4b5 2023-04-05 04:57:24,176 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/be5f13bb192c4555b0627c2187c25102 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/be5f13bb192c4555b0627c2187c25102 2023-04-05 04:57:24,199 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/be5f13bb192c4555b0627c2187c25102, entries=200, sequenceid=318, filesize=14.3 K 2023-04-05 04:57:24,200 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/a57987ebd5384b15a1a327ff0dc87907 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/a57987ebd5384b15a1a327ff0dc87907 2023-04-05 04:57:24,207 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/a57987ebd5384b15a1a327ff0dc87907, entries=150, sequenceid=318, filesize=11.9 K 2023-04-05 04:57:24,210 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/69c6760c7d2a48dbab37dd28ddb3c4b5 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/69c6760c7d2a48dbab37dd28ddb3c4b5 2023-04-05 04:57:24,227 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/69c6760c7d2a48dbab37dd28ddb3c4b5, entries=150, sequenceid=318, filesize=11.9 K 2023-04-05 04:57:24,228 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for d690627297fb3f579eb1d6c5d71ead3a in 249ms, sequenceid=318, compaction requested=true 2023-04-05 04:57:24,228 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:24,228 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:57:24,229 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:57:24,229 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:57:24,229 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:57:24,232 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:57:24,234 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:24,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:24,236 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49038 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:57:24,236 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): d690627297fb3f579eb1d6c5d71ead3a/B is initiating minor compaction (all files) 2023-04-05 04:57:24,236 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d690627297fb3f579eb1d6c5d71ead3a/B in TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:24,236 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/f3a0d93bfaa8444eae4a6d4376251b62, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/d1af6161321740ec9f5e55609266fc36, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/76fad2de8c5b43a8a6050baa5ce78411, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/a57987ebd5384b15a1a327ff0dc87907] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp, totalSize=47.9 K 2023-04-05 04:57:24,237 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51478 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:57:24,238 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): d690627297fb3f579eb1d6c5d71ead3a/A is initiating minor compaction (all files) 2023-04-05 04:57:24,238 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d690627297fb3f579eb1d6c5d71ead3a/A in TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:24,238 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/adec1d8788274ffb908c2021ec4fb471, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/b30329d40e2349a48fe9dffd5a0a963d, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/9b12e5385102444a978e2d586bc00dca, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/be5f13bb192c4555b0627c2187c25102] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp, totalSize=50.3 K 2023-04-05 04:57:24,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:24,239 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting f3a0d93bfaa8444eae4a6d4376251b62, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1680670641481 2023-04-05 04:57:24,240 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting d1af6161321740ec9f5e55609266fc36, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1680670641532 2023-04-05 04:57:24,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:24,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:24,241 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:24,241 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting adec1d8788274ffb908c2021ec4fb471, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1680670641481 2023-04-05 04:57:24,241 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:24,241 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:24,241 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:24,241 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 76fad2de8c5b43a8a6050baa5ce78411, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1680670642124 2023-04-05 04:57:24,241 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting b30329d40e2349a48fe9dffd5a0a963d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1680670641532 2023-04-05 04:57:24,241 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:24,241 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:24,241 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting a57987ebd5384b15a1a327ff0dc87907, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1680670643340 2023-04-05 04:57:24,241 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 9b12e5385102444a978e2d586bc00dca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1680670642124 2023-04-05 04:57:24,242 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting be5f13bb192c4555b0627c2187c25102, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1680670643340 2023-04-05 04:57:24,242 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:24,242 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:24,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:24,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:24,242 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:24,242 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:24,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:24,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:24,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:24,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:24,243 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:24,258 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:24,258 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:24,258 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:24,258 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:24,268 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): d690627297fb3f579eb1d6c5d71ead3a#A#compaction#242 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:57:24,278 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): d690627297fb3f579eb1d6c5d71ead3a#B#compaction#243 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:57:24,288 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:24,288 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:24,293 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:24,293 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:24,293 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d690627297fb3f579eb1d6c5d71ead3a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 04:57:24,293 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=A 2023-04-05 04:57:24,293 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:24,293 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=B 2023-04-05 04:57:24,293 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:24,293 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=C 2023-04-05 04:57:24,294 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:24,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on d690627297fb3f579eb1d6c5d71ead3a 2023-04-05 04:57:24,320 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:24,342 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/6700a2ca8c8744d2b01a6baae69c046b as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/6700a2ca8c8744d2b01a6baae69c046b 2023-04-05 04:57:24,348 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d690627297fb3f579eb1d6c5d71ead3a/A of d690627297fb3f579eb1d6c5d71ead3a into 6700a2ca8c8744d2b01a6baae69c046b(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:57:24,348 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:24,348 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a., storeName=d690627297fb3f579eb1d6c5d71ead3a/A, priority=12, startTime=1680670644228; duration=0sec 2023-04-05 04:57:24,349 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:57:24,349 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:57:24,350 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49038 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:57:24,350 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): d690627297fb3f579eb1d6c5d71ead3a/C is initiating minor compaction (all files) 2023-04-05 04:57:24,351 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d690627297fb3f579eb1d6c5d71ead3a/C in TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:24,351 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/42ab4f5e223d4d1ca3ef2f12b4ce83dc, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/8d6fb8800aa24e4996d4b52ff0269775, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/f5c062b08845408cab8418b68105784c, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/69c6760c7d2a48dbab37dd28ddb3c4b5] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp, totalSize=47.9 K 2023-04-05 04:57:24,351 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 42ab4f5e223d4d1ca3ef2f12b4ce83dc, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1680670641481 2023-04-05 04:57:24,352 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 8d6fb8800aa24e4996d4b52ff0269775, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1680670641532 2023-04-05 04:57:24,353 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting f5c062b08845408cab8418b68105784c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1680670642124 2023-04-05 04:57:24,353 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 69c6760c7d2a48dbab37dd28ddb3c4b5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1680670643340 2023-04-05 04:57:24,376 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): d690627297fb3f579eb1d6c5d71ead3a#C#compaction#245 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:57:24,406 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=328 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/6a80bc29688345d1b7962755b65f3fb3 2023-04-05 04:57:24,419 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670644348, max timestamp=1680670644416Num uniques 859; 2023-04-05 04:57:24,419 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:57:24,419 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670644348, max timestamp=1680670644416Num uniques 859; 2023-04-05 04:57:24,419 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670644348, max timestamp=1680670644416Num uniques 859; 2023-04-05 04:57:24,419 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:57:24,419 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:57:24,490 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:24,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41488 deadline: 1680670704488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:24,491 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:24,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41420 deadline: 1680670704489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:24,596 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:24,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41420 deadline: 1680670704593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:24,596 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:24,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41488 deadline: 1680670704593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:24,764 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/e02f4d150763452fbd217ec62f52bdb9 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/e02f4d150763452fbd217ec62f52bdb9 2023-04-05 04:57:24,770 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d690627297fb3f579eb1d6c5d71ead3a/B of d690627297fb3f579eb1d6c5d71ead3a into e02f4d150763452fbd217ec62f52bdb9(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:57:24,770 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:24,770 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a., storeName=d690627297fb3f579eb1d6c5d71ead3a/B, priority=12, startTime=1680670644229; duration=0sec 2023-04-05 04:57:24,770 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:57:24,801 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:24,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41420 deadline: 1680670704798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:24,802 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:24,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41488 deadline: 1680670704798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:24,831 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/c5b77dbfc23648589ad4bfe582894d85 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/c5b77dbfc23648589ad4bfe582894d85 2023-04-05 04:57:24,837 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d690627297fb3f579eb1d6c5d71ead3a/C of d690627297fb3f579eb1d6c5d71ead3a into c5b77dbfc23648589ad4bfe582894d85(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:57:24,837 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:24,837 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a., storeName=d690627297fb3f579eb1d6c5d71ead3a/C, priority=12, startTime=1680670644232; duration=0sec 2023-04-05 04:57:24,837 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:57:24,885 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=328 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/0a1cfc48f53f46bd8d1489a3a4b22490 2023-04-05 04:57:25,105 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:25,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41420 deadline: 1680670705102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:25,106 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:25,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41488 deadline: 1680670705105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:25,345 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=328 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/32f0fbeede894e8a94cd35090cf0200b 2023-04-05 04:57:25,349 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/6a80bc29688345d1b7962755b65f3fb3 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/6a80bc29688345d1b7962755b65f3fb3 2023-04-05 04:57:25,354 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/6a80bc29688345d1b7962755b65f3fb3, entries=150, sequenceid=328, filesize=11.9 K 2023-04-05 04:57:25,354 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/0a1cfc48f53f46bd8d1489a3a4b22490 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/0a1cfc48f53f46bd8d1489a3a4b22490 2023-04-05 04:57:25,360 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/0a1cfc48f53f46bd8d1489a3a4b22490, entries=150, sequenceid=328, filesize=11.9 K 2023-04-05 04:57:25,361 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/32f0fbeede894e8a94cd35090cf0200b as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/32f0fbeede894e8a94cd35090cf0200b 2023-04-05 04:57:25,365 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/32f0fbeede894e8a94cd35090cf0200b, entries=150, sequenceid=328, filesize=11.9 K 2023-04-05 04:57:25,366 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for d690627297fb3f579eb1d6c5d71ead3a in 1073ms, sequenceid=328, compaction requested=false 2023-04-05 04:57:25,366 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:25,366 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:25,366 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:25,366 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:25,366 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:25,366 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:25,366 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:25,368 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:25,368 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:25,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:25,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:25,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:25,368 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:25,368 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:25,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:25,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:25,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:25,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:25,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:25,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:25,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4710d0b4[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:25,370 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:25,371 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:25,392 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:25,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:25,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:25,393 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:25,393 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:25,398 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:25,398 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:25,398 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:25,420 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:25,420 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:25,420 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:25,422 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:25,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:25,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:25,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:25,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:25,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:25,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:25,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:25,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:25,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:25,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:25,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:25,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:25,428 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:25,428 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:25,428 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:25,428 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:25,428 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:25,429 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:25,429 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:25,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:25,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:25,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:25,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:25,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:25,430 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:25,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:25,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:25,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:25,430 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:25,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:25,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:25,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:25,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:25,431 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:25,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:25,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:25,436 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:25,436 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:25,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:25,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:25,436 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:25,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:25,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:25,436 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:25,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:25,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:25,436 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:25,437 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:25,437 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:25,437 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:25,437 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:25,437 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:25,437 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:25,437 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:25,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:25,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:25,536 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:25,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:25,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:25,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:25,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:25,544 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:25,545 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:25,545 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:25,545 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:25,546 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:25,546 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:25,547 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:25,547 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:25,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:25,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:25,547 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:25,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:25,547 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:25,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:25,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:25,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:25,548 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:25,548 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:25,548 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:25,548 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:25,548 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:25,548 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:25,548 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:25,548 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:25,548 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:25,548 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:25,548 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d690627297fb3f579eb1d6c5d71ead3a 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-05 04:57:25,548 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=A 2023-04-05 04:57:25,549 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:25,549 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=B 2023-04-05 04:57:25,549 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:25,549 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=C 2023-04-05 04:57:25,549 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:25,580 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/cb3b7540a979429bad8205005206675a 2023-04-05 04:57:25,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on d690627297fb3f579eb1d6c5d71ead3a 2023-04-05 04:57:25,611 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:25,634 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:25,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41488 deadline: 1680670705631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:25,635 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:25,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41420 deadline: 1680670705632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:25,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:25,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1680670705669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:25,675 DEBUG [Thread-1610] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4137 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a., hostname=jenkins-hbase20.apache.org,38091,1680670569206, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 04:57:25,688 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:25,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41454 deadline: 1680670705687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:25,691 DEBUG [Thread-1616] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4153 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a., hostname=jenkins-hbase20.apache.org,38091,1680670569206, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 04:57:25,737 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:25,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41488 deadline: 1680670705735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:25,737 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:25,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41420 deadline: 1680670705736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:25,944 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:25,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41420 deadline: 1680670705942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:25,944 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:25,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41488 deadline: 1680670705942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:26,019 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/8335910e42c0404e87ce76f7146447bd 2023-04-05 04:57:26,247 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:26,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41488 deadline: 1680670706245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:26,250 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:26,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41420 deadline: 1680670706250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:26,443 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/c084d15e57814736bf93bc25f4d75b48 2023-04-05 04:57:26,447 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/cb3b7540a979429bad8205005206675a as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/cb3b7540a979429bad8205005206675a 2023-04-05 04:57:26,451 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/cb3b7540a979429bad8205005206675a, entries=150, sequenceid=367, filesize=11.9 K 2023-04-05 04:57:26,451 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/8335910e42c0404e87ce76f7146447bd as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/8335910e42c0404e87ce76f7146447bd 2023-04-05 04:57:26,454 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/8335910e42c0404e87ce76f7146447bd, entries=150, sequenceid=367, filesize=11.9 K 2023-04-05 04:57:26,455 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/c084d15e57814736bf93bc25f4d75b48 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/c084d15e57814736bf93bc25f4d75b48 2023-04-05 04:57:26,463 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/c084d15e57814736bf93bc25f4d75b48, entries=150, sequenceid=367, filesize=11.9 K 2023-04-05 04:57:26,464 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=53.67 KB/54960 for d690627297fb3f579eb1d6c5d71ead3a in 916ms, sequenceid=367, compaction requested=true 2023-04-05 04:57:26,464 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:26,464 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:26,464 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:26,465 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:26,465 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:26,465 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:26,465 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:26,466 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:26,466 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:26,466 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:26,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:26,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:26,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:26,466 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:26,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:26,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:26,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:26,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:26,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:26,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:26,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@321afd88[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:26,478 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:26,478 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:26,483 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:26,483 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:26,493 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:26,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:26,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:26,493 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:26,493 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:26,493 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:26,494 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:26,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:26,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:26,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:26,494 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:26,494 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:26,494 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:26,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:26,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:26,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:26,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:26,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:26,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:26,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:26,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:26,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:26,503 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:26,503 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:26,503 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:26,503 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:26,503 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:26,508 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:26,508 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:26,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:26,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:26,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:26,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:26,508 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:26,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:26,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:26,512 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:26,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:26,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:26,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:26,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:26,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:26,513 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:26,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:26,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:26,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:26,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:26,516 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:26,516 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:26,516 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:26,516 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:26,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:26,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:26,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:26,516 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:26,517 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:26,517 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:26,517 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:26,517 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:26,517 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:26,519 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:26,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:26,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:26,617 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:26,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:26,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:26,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:26,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:26,633 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:26,633 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:26,633 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:26,633 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:26,633 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:26,633 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:26,633 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:26,634 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:26,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:26,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:26,634 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:26,634 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:26,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:26,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:26,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:26,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:26,634 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:26,637 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:26,637 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:26,637 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:26,637 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:26,637 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:26,638 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:26,638 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:26,638 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:26,638 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:26,639 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d690627297fb3f579eb1d6c5d71ead3a 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 04:57:26,639 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=A 2023-04-05 04:57:26,639 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:26,639 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=B 2023-04-05 04:57:26,639 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:26,639 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=C 2023-04-05 04:57:26,639 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:26,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on d690627297fb3f579eb1d6c5d71ead3a 2023-04-05 04:57:26,753 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:26,795 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670646783, max timestamp=1680670646793Num uniques 859; 2023-04-05 04:57:26,795 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:57:26,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670646783, max timestamp=1680670646793Num uniques 859; 2023-04-05 04:57:26,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:57:26,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670646783, max timestamp=1680670646793Num uniques 859; 2023-04-05 04:57:26,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:57:26,837 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:26,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41488 deadline: 1680670706832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:26,839 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:26,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41420 deadline: 1680670706837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:26,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:26,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41488 deadline: 1680670706938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:26,948 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:26,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41420 deadline: 1680670706945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:27,086 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=378 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/3a2201b280694da7aff1214308b02e33 2023-04-05 04:57:27,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:27,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41488 deadline: 1680670707146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:27,156 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:27,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41420 deadline: 1680670707152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:27,452 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:27,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41488 deadline: 1680670707449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:27,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:27,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41420 deadline: 1680670707458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:27,508 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=378 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/40004b38dadd4b00a5a2afed685b8e51 2023-04-05 04:57:27,552 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=378 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/ffc7cb5d86094100a3e96f39c713d0c0 2023-04-05 04:57:27,559 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/3a2201b280694da7aff1214308b02e33 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/3a2201b280694da7aff1214308b02e33 2023-04-05 04:57:27,563 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/3a2201b280694da7aff1214308b02e33, entries=150, sequenceid=378, filesize=11.9 K 2023-04-05 04:57:27,566 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/40004b38dadd4b00a5a2afed685b8e51 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/40004b38dadd4b00a5a2afed685b8e51 2023-04-05 04:57:27,571 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/40004b38dadd4b00a5a2afed685b8e51, entries=150, sequenceid=378, filesize=11.9 K 2023-04-05 04:57:27,572 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/ffc7cb5d86094100a3e96f39c713d0c0 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/ffc7cb5d86094100a3e96f39c713d0c0 2023-04-05 04:57:27,575 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/ffc7cb5d86094100a3e96f39c713d0c0, entries=150, sequenceid=378, filesize=11.9 K 2023-04-05 04:57:27,576 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for d690627297fb3f579eb1d6c5d71ead3a in 937ms, sequenceid=378, compaction requested=true 2023-04-05 04:57:27,576 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:27,576 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:27,577 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:27,577 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:27,577 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:27,577 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:27,577 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:27,580 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:27,580 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:27,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:27,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:27,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:27,580 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:27,580 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:27,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:27,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:27,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:27,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:27,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:27,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:27,581 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:27,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@195310c3[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:27,581 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:27,582 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:27,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:27,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:27,582 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:27,582 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:27,582 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:27,582 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:27,582 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:27,584 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:27,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:27,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:27,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:27,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:27,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:27,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:27,585 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:27,585 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:27,585 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:27,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:27,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:27,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:27,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:27,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:27,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:27,604 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:27,604 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:27,604 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:27,604 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:27,604 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:27,606 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:27,606 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:27,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:27,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:27,606 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:27,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:27,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:27,607 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:27,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:27,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:27,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:27,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:27,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:27,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:27,616 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:27,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:27,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:27,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:27,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:27,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:27,618 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:27,618 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:27,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:27,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:27,618 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:27,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:27,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:27,619 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:27,619 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:27,618 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:27,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:27,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:27,619 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:27,619 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:27,619 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:27,619 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:27,619 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:27,619 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:27,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:27,719 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:27,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:27,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:27,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:27,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:27,726 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:27,726 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:27,726 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:27,726 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:27,730 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:27,730 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:27,734 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:27,734 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:27,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:27,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:27,735 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:27,735 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:27,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:27,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:27,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:27,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:27,739 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:27,740 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:27,740 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:27,740 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:27,740 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:27,740 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:27,742 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:27,742 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:27,742 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:27,742 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:27,742 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d690627297fb3f579eb1d6c5d71ead3a 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-05 04:57:27,744 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=A 2023-04-05 04:57:27,744 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:27,744 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=B 2023-04-05 04:57:27,744 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:27,744 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=C 2023-04-05 04:57:27,744 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:27,772 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=414 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/a0b8aacb12dc4336bb024ac62285a3d8 2023-04-05 04:57:27,799 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=414 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/edc1a6a7aaab4f06b68076de9c507a2f 2023-04-05 04:57:27,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on d690627297fb3f579eb1d6c5d71ead3a 2023-04-05 04:57:27,824 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:27,831 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=414 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/50c03d0a649b4f699bef2c465c21b5e3 2023-04-05 04:57:27,841 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/a0b8aacb12dc4336bb024ac62285a3d8 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/a0b8aacb12dc4336bb024ac62285a3d8 2023-04-05 04:57:27,851 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/a0b8aacb12dc4336bb024ac62285a3d8, entries=150, sequenceid=414, filesize=11.9 K 2023-04-05 04:57:27,851 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/edc1a6a7aaab4f06b68076de9c507a2f as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/edc1a6a7aaab4f06b68076de9c507a2f 2023-04-05 04:57:27,855 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/edc1a6a7aaab4f06b68076de9c507a2f, entries=150, sequenceid=414, filesize=11.9 K 2023-04-05 04:57:27,855 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/50c03d0a649b4f699bef2c465c21b5e3 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/50c03d0a649b4f699bef2c465c21b5e3 2023-04-05 04:57:27,867 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/50c03d0a649b4f699bef2c465c21b5e3, entries=150, sequenceid=414, filesize=11.9 K 2023-04-05 04:57:27,868 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for d690627297fb3f579eb1d6c5d71ead3a in 126ms, sequenceid=414, compaction requested=true 2023-04-05 04:57:27,868 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:27,868 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:27,868 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:27,868 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:27,868 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:27,871 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:27,871 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:27,872 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:27,872 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:27,872 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:27,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on d690627297fb3f579eb1d6c5d71ead3a 2023-04-05 04:57:27,874 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d690627297fb3f579eb1d6c5d71ead3a 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-05 04:57:27,875 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=A 2023-04-05 04:57:27,875 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:27,875 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=B 2023-04-05 04:57:27,875 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:27,875 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=C 2023-04-05 04:57:27,875 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:27,876 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:27,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:27,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:27,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:27,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:27,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:27,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:27,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:27,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:27,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:27,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e4b2361[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:27,878 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:27,879 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:27,885 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:27,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:27,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:27,886 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:27,886 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:27,886 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:27,890 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:27,890 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:27,890 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:27,890 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:27,891 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:27,891 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:27,891 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:27,891 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:27,891 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:27,891 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:27,891 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:27,892 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:27,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:27,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:27,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:27,892 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:27,892 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:27,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:27,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:27,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:27,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:27,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:27,893 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:27,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:27,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:27,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:27,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:27,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:27,894 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:27,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:27,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:27,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:27,896 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:27,896 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:27,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:27,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:27,896 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:27,896 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:27,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:27,896 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:27,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:27,896 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:27,896 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:27,896 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:27,896 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:27,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:27,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:27,896 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:27,896 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:27,896 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:27,969 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:27,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41420 deadline: 1680670707964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:27,969 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:27,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41488 deadline: 1680670707965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:27,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:27,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41430 deadline: 1680670707967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:27,996 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:27,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:28,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:28,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:28,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:28,005 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:28,005 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:28,005 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:28,005 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:28,007 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:28,007 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:28,007 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:28,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:28,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:28,007 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:28,008 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,008 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:28,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:28,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:28,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:28,009 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:28,012 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:28,012 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:28,012 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:28,012 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:28,012 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:28,014 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:28,014 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:28,014 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:28,014 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:28,015 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:28,015 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:28,017 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:28,017 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:28,017 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,017 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:28,017 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:28,022 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,022 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:28,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:28,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:28,023 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:28,023 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:28,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:28,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:28,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:28,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:28,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:28,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f9fd56c[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:28,026 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:28,026 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:28,031 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:28,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:28,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:28,031 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:28,031 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:28,031 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:28,035 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,036 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:28,036 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:28,036 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:28,036 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:28,036 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:28,036 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:28,036 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:28,037 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:28,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:28,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:28,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:28,038 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:28,038 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:28,038 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:28,038 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:28,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:28,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:28,038 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:28,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:28,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:28,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:28,040 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:28,041 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:28,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:28,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:28,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:28,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:28,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,043 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:28,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:28,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:28,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:28,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:28,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:28,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:28,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:28,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:28,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:28,044 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:28,044 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:28,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:28,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:28,045 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:28,073 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:28,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41430 deadline: 1680670708071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,073 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:28,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41488 deadline: 1680670708072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,144 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:28,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:28,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:28,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:28,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:28,157 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:28,157 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:28,157 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:28,157 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:28,157 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:28,157 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:28,158 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,158 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,158 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:28,160 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:28,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:28,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:28,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:28,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:28,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:28,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:28,162 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:28,163 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:28,163 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:28,163 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:28,163 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:28,163 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:28,165 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:28,165 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:28,165 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:28,165 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:28,165 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:28,165 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:28,165 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:28,165 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:28,165 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,165 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:28,165 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:28,172 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:28,172 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:28,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:28,173 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:28,173 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:28,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:28,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:28,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:28,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:28,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:28,187 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:28,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a8c85aa[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:28,187 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:28,187 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:28,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:28,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:28,188 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:28,188 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:28,188 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:28,188 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,188 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:28,189 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,189 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:28,189 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:28,189 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:28,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:28,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:28,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:28,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:28,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:28,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:28,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:28,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:28,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:28,196 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:28,196 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:28,196 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:28,196 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:28,196 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:28,199 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:28,205 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:28,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:28,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,200 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:28,210 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:28,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:28,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:28,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:28,215 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,217 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,217 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:28,217 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:28,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:28,217 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:28,217 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:28,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:28,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:28,217 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:28,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:28,217 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,217 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:28,217 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:28,217 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:28,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:28,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:28,219 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:28,219 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:28,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:28,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41488 deadline: 1680670708275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:28,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41430 deadline: 1680670708275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,305 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=426 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/777c55cebd764a97b86a5985234c14ad 2023-04-05 04:57:28,319 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:28,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:28,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:28,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:28,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:28,323 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:28,323 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:28,323 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:28,323 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:28,323 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:28,323 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:28,324 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:28,324 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:28,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:28,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:28,325 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,325 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:28,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:28,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:28,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:28,325 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:28,325 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:28,325 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:28,325 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:28,326 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:28,326 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:28,331 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:28,331 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:28,332 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:28,332 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:28,332 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=426 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/ab5097a1a60d4ce69e0aa02d8c54776a 2023-04-05 04:57:28,332 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:28,332 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:28,332 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:28,332 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:28,332 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,332 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:28,332 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:28,335 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,335 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:28,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:28,336 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:28,336 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:28,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:28,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:28,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:28,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:28,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:28,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bd85f30[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:28,347 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:28,347 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:28,348 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:28,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:28,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:28,348 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:28,348 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:28,348 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:28,348 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:28,348 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:28,349 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:28,349 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,349 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:28,349 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:28,349 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:28,349 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:28,349 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:28,349 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:28,349 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:28,350 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:28,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:28,350 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:28,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:28,350 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:28,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:28,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:28,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:28,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:28,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:28,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:28,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:28,350 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:28,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:28,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:28,352 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,352 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=426 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/5cb7557e1da8445db174319da25ad558 2023-04-05 04:57:28,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:28,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:28,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,355 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,355 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:28,355 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:28,355 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,356 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:28,356 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:28,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:28,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:28,356 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:28,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:28,356 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:28,356 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:28,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:28,356 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:28,356 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:28,356 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:28,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:28,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:28,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:28,357 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/777c55cebd764a97b86a5985234c14ad as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/777c55cebd764a97b86a5985234c14ad 2023-04-05 04:57:28,374 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/777c55cebd764a97b86a5985234c14ad, entries=200, sequenceid=426, filesize=14.3 K 2023-04-05 04:57:28,377 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/ab5097a1a60d4ce69e0aa02d8c54776a as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/ab5097a1a60d4ce69e0aa02d8c54776a 2023-04-05 04:57:28,381 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/ab5097a1a60d4ce69e0aa02d8c54776a, entries=150, sequenceid=426, filesize=11.9 K 2023-04-05 04:57:28,382 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/5cb7557e1da8445db174319da25ad558 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/5cb7557e1da8445db174319da25ad558 2023-04-05 04:57:28,385 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/5cb7557e1da8445db174319da25ad558, entries=150, sequenceid=426, filesize=11.9 K 2023-04-05 04:57:28,386 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for d690627297fb3f579eb1d6c5d71ead3a in 512ms, sequenceid=426, compaction requested=true 2023-04-05 04:57:28,386 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:28,386 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:57:28,386 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-05 04:57:28,386 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:57:28,386 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-05 04:57:28,386 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:57:28,405 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 76136 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-05 04:57:28,405 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): d690627297fb3f579eb1d6c5d71ead3a/A is initiating minor compaction (all files) 2023-04-05 04:57:28,405 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d690627297fb3f579eb1d6c5d71ead3a/A in TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:28,405 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/6700a2ca8c8744d2b01a6baae69c046b, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/6a80bc29688345d1b7962755b65f3fb3, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/cb3b7540a979429bad8205005206675a, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/3a2201b280694da7aff1214308b02e33, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/a0b8aacb12dc4336bb024ac62285a3d8, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/777c55cebd764a97b86a5985234c14ad] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp, totalSize=74.4 K 2023-04-05 04:57:28,406 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73696 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-05 04:57:28,406 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): d690627297fb3f579eb1d6c5d71ead3a/B is initiating minor compaction (all files) 2023-04-05 04:57:28,406 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d690627297fb3f579eb1d6c5d71ead3a/B in TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:28,406 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/e02f4d150763452fbd217ec62f52bdb9, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/0a1cfc48f53f46bd8d1489a3a4b22490, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/8335910e42c0404e87ce76f7146447bd, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/40004b38dadd4b00a5a2afed685b8e51, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/edc1a6a7aaab4f06b68076de9c507a2f, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/ab5097a1a60d4ce69e0aa02d8c54776a] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp, totalSize=72.0 K 2023-04-05 04:57:28,407 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 6700a2ca8c8744d2b01a6baae69c046b, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1680670643340 2023-04-05 04:57:28,407 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 6a80bc29688345d1b7962755b65f3fb3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=328, earliestPutTs=1680670643998 2023-04-05 04:57:28,407 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting e02f4d150763452fbd217ec62f52bdb9, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1680670643340 2023-04-05 04:57:28,407 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting cb3b7540a979429bad8205005206675a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1680670644381 2023-04-05 04:57:28,408 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 0a1cfc48f53f46bd8d1489a3a4b22490, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=328, earliestPutTs=1680670643998 2023-04-05 04:57:28,408 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 3a2201b280694da7aff1214308b02e33, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=378, earliestPutTs=1680670645609 2023-04-05 04:57:28,408 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 8335910e42c0404e87ce76f7146447bd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1680670644381 2023-04-05 04:57:28,409 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting a0b8aacb12dc4336bb024ac62285a3d8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=414, earliestPutTs=1680670646807 2023-04-05 04:57:28,409 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 40004b38dadd4b00a5a2afed685b8e51, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=378, earliestPutTs=1680670645609 2023-04-05 04:57:28,409 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 777c55cebd764a97b86a5985234c14ad, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1680670647853 2023-04-05 04:57:28,409 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting edc1a6a7aaab4f06b68076de9c507a2f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=414, earliestPutTs=1680670646807 2023-04-05 04:57:28,410 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting ab5097a1a60d4ce69e0aa02d8c54776a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1680670647857 2023-04-05 04:57:28,449 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): d690627297fb3f579eb1d6c5d71ead3a#B#compaction#260 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:57:28,453 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): d690627297fb3f579eb1d6c5d71ead3a#A#compaction#261 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:57:28,456 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:28,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:28,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:28,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:28,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:28,470 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:28,471 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:28,471 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:28,471 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:28,471 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:28,471 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:28,472 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:28,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:28,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:28,472 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,472 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,472 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:28,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:28,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:28,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:28,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:28,473 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:28,473 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:28,473 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:28,473 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:28,473 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:28,473 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:28,474 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:28,474 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:28,474 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:28,474 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:28,474 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d690627297fb3f579eb1d6c5d71ead3a 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-05 04:57:28,474 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=A 2023-04-05 04:57:28,474 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:28,474 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=B 2023-04-05 04:57:28,474 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:28,474 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=C 2023-04-05 04:57:28,474 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:28,475 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/cde7374cfdfa48238b9c96075fa15596 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/cde7374cfdfa48238b9c96075fa15596 2023-04-05 04:57:28,479 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in d690627297fb3f579eb1d6c5d71ead3a/B of d690627297fb3f579eb1d6c5d71ead3a into cde7374cfdfa48238b9c96075fa15596(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:57:28,479 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:28,479 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a., storeName=d690627297fb3f579eb1d6c5d71ead3a/B, priority=10, startTime=1680670648386; duration=0sec 2023-04-05 04:57:28,479 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:57:28,479 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-05 04:57:28,482 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73696 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-05 04:57:28,482 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): d690627297fb3f579eb1d6c5d71ead3a/C is initiating minor compaction (all files) 2023-04-05 04:57:28,482 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d690627297fb3f579eb1d6c5d71ead3a/C in TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:28,482 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/c5b77dbfc23648589ad4bfe582894d85, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/32f0fbeede894e8a94cd35090cf0200b, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/c084d15e57814736bf93bc25f4d75b48, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/ffc7cb5d86094100a3e96f39c713d0c0, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/50c03d0a649b4f699bef2c465c21b5e3, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/5cb7557e1da8445db174319da25ad558] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp, totalSize=72.0 K 2023-04-05 04:57:28,482 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting c5b77dbfc23648589ad4bfe582894d85, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1680670643340 2023-04-05 04:57:28,483 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 32f0fbeede894e8a94cd35090cf0200b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=328, earliestPutTs=1680670643998 2023-04-05 04:57:28,483 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting c084d15e57814736bf93bc25f4d75b48, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1680670644381 2023-04-05 04:57:28,484 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting ffc7cb5d86094100a3e96f39c713d0c0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=378, earliestPutTs=1680670645609 2023-04-05 04:57:28,484 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 50c03d0a649b4f699bef2c465c21b5e3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=414, earliestPutTs=1680670646807 2023-04-05 04:57:28,484 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 5cb7557e1da8445db174319da25ad558, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1680670647857 2023-04-05 04:57:28,508 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): d690627297fb3f579eb1d6c5d71ead3a#C#compaction#263 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:57:28,534 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/e7360a41ceeb4747a6aa2121ba1fb83d as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/e7360a41ceeb4747a6aa2121ba1fb83d 2023-04-05 04:57:28,551 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in d690627297fb3f579eb1d6c5d71ead3a/C of d690627297fb3f579eb1d6c5d71ead3a into e7360a41ceeb4747a6aa2121ba1fb83d(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:57:28,551 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:28,551 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a., storeName=d690627297fb3f579eb1d6c5d71ead3a/C, priority=10, startTime=1680670648386; duration=0sec 2023-04-05 04:57:28,551 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:57:28,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on d690627297fb3f579eb1d6c5d71ead3a 2023-04-05 04:57:28,583 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:28,599 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:28,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41488 deadline: 1680670708599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,603 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:28,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41430 deadline: 1680670708602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,702 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:28,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41488 deadline: 1680670708700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,707 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:28,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41430 deadline: 1680670708706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,922 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:28,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41430 deadline: 1680670708920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,922 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:28,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41488 deadline: 1680670708921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:28,924 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/fe2edbde9dd84addb90eceb2909ad773 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/fe2edbde9dd84addb90eceb2909ad773 2023-04-05 04:57:28,926 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=446 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/988569b968af41f7bdbc5bc538f7d9e1 2023-04-05 04:57:28,933 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in d690627297fb3f579eb1d6c5d71ead3a/A of d690627297fb3f579eb1d6c5d71ead3a into fe2edbde9dd84addb90eceb2909ad773(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:57:28,933 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:28,933 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a., storeName=d690627297fb3f579eb1d6c5d71ead3a/A, priority=10, startTime=1680670648386; duration=0sec 2023-04-05 04:57:28,933 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:57:28,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:28,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41420 deadline: 1680670708979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:29,227 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:29,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41430 deadline: 1680670709225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:29,228 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:29,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41488 deadline: 1680670709227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:29,351 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=446 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/1a87b4e4ebcc476c9ee046301edeb0b9 2023-04-05 04:57:29,371 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=446 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/9aeed6a50c6d496b8505720df74a72e3 2023-04-05 04:57:29,383 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/988569b968af41f7bdbc5bc538f7d9e1 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/988569b968af41f7bdbc5bc538f7d9e1 2023-04-05 04:57:29,389 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/988569b968af41f7bdbc5bc538f7d9e1, entries=150, sequenceid=446, filesize=11.9 K 2023-04-05 04:57:29,394 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/1a87b4e4ebcc476c9ee046301edeb0b9 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/1a87b4e4ebcc476c9ee046301edeb0b9 2023-04-05 04:57:29,397 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/1a87b4e4ebcc476c9ee046301edeb0b9, entries=150, sequenceid=446, filesize=11.9 K 2023-04-05 04:57:29,398 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/9aeed6a50c6d496b8505720df74a72e3 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/9aeed6a50c6d496b8505720df74a72e3 2023-04-05 04:57:29,401 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/9aeed6a50c6d496b8505720df74a72e3, entries=150, sequenceid=446, filesize=11.9 K 2023-04-05 04:57:29,402 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for d690627297fb3f579eb1d6c5d71ead3a in 928ms, sequenceid=446, compaction requested=false 2023-04-05 04:57:29,402 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:29,402 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:29,409 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:29,409 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:29,409 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:29,409 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:29,409 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:29,410 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:29,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:29,410 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:29,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:29,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:29,411 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:29,411 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:29,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:29,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:29,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:29,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:29,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:29,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:29,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c4f3301[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:29,412 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:29,412 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:29,412 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:29,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:29,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:29,412 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:29,412 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:29,412 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:29,413 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:29,413 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:29,413 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:29,413 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:29,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:29,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:29,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:29,413 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:29,413 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:29,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:29,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:29,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:29,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:29,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:29,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:29,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:29,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:29,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:29,419 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:29,419 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:29,419 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:29,419 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:29,420 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:29,420 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:29,420 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:29,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:29,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:29,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:29,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:29,421 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:29,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:29,423 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:29,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:29,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:29,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:29,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:29,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:29,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:29,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:29,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:29,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:29,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:29,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:29,426 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:29,435 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:29,435 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:29,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:29,435 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:29,435 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:29,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:29,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:29,436 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:29,436 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:29,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:29,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:29,436 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:29,436 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:29,436 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:29,436 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:29,436 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:29,436 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:29,536 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:29,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:29,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:29,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:29,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:29,544 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:29,545 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:29,545 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:29,545 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:29,551 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:29,551 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:29,557 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:29,557 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:29,557 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:29,558 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:29,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:29,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:29,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:29,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:29,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:29,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:29,565 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:29,565 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:29,565 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:29,565 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:29,565 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:29,565 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:29,568 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:29,568 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:29,569 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:29,569 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:29,569 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d690627297fb3f579eb1d6c5d71ead3a 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-05 04:57:29,570 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=A 2023-04-05 04:57:29,570 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:29,570 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=B 2023-04-05 04:57:29,570 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:29,570 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=C 2023-04-05 04:57:29,570 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:29,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on d690627297fb3f579eb1d6c5d71ead3a 2023-04-05 04:57:29,685 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:29,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:29,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1680670709733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:29,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:29,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41488 deadline: 1680670709733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:29,740 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:29,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41454 deadline: 1680670709737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:29,740 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:29,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41430 deadline: 1680670709738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:29,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:29,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1680670709838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:29,842 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:29,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41454 deadline: 1680670709841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:29,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:29,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41488 deadline: 1680670709846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:30,043 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/c43833756b72428a88662f6b34bc3ed9 2023-04-05 04:57:30,047 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:30,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1680670710045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:30,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:30,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41454 deadline: 1680670710047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:30,051 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:30,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41488 deadline: 1680670710048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:30,350 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:30,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1680670710349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:30,350 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:30,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41454 deadline: 1680670710349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:30,357 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:30,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41488 deadline: 1680670710355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:30,484 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/dac6339a6350410bb7e0019e0df2ecb0 2023-04-05 04:57:30,506 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/4a9eec3c96b14290b7d4c5fe71a83cc1 2023-04-05 04:57:30,512 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/c43833756b72428a88662f6b34bc3ed9 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/c43833756b72428a88662f6b34bc3ed9 2023-04-05 04:57:30,516 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/c43833756b72428a88662f6b34bc3ed9, entries=150, sequenceid=461, filesize=11.9 K 2023-04-05 04:57:30,517 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/dac6339a6350410bb7e0019e0df2ecb0 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/dac6339a6350410bb7e0019e0df2ecb0 2023-04-05 04:57:30,524 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/dac6339a6350410bb7e0019e0df2ecb0, entries=150, sequenceid=461, filesize=11.9 K 2023-04-05 04:57:30,525 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/4a9eec3c96b14290b7d4c5fe71a83cc1 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/4a9eec3c96b14290b7d4c5fe71a83cc1 2023-04-05 04:57:30,534 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/4a9eec3c96b14290b7d4c5fe71a83cc1, entries=150, sequenceid=461, filesize=11.9 K 2023-04-05 04:57:30,535 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for d690627297fb3f579eb1d6c5d71ead3a in 966ms, sequenceid=461, compaction requested=true 2023-04-05 04:57:30,535 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:30,535 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:30,535 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:30,535 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:30,535 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:30,541 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:30,541 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:30,542 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:30,542 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:30,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:30,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:30,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:30,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:30,542 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:30,543 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:30,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:30,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:30,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:30,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:30,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:30,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d5abd51[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:30,544 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:30,544 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:30,544 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:30,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:30,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:30,544 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:30,544 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:30,544 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:30,544 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:30,544 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:30,545 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:30,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:30,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:30,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:30,545 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:30,545 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:30,545 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:30,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:30,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:30,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:30,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:30,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:30,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:30,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:30,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:30,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:30,547 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:30,547 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:30,547 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:30,547 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:30,547 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:30,548 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:30,548 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:30,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:30,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:30,548 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:30,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:30,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:30,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:30,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:30,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:30,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:30,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:30,549 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:30,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:30,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:30,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:30,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:30,550 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:30,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:30,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:30,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:30,553 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:30,553 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:30,553 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:30,553 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:30,553 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:30,553 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:30,553 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:30,553 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:30,553 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:30,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:30,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:30,553 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:30,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:30,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:30,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:30,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:30,554 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:30,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:30,554 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:30,654 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:30,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:30,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:30,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:30,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:30,657 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:30,657 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:30,657 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:30,657 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:30,666 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:30,666 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:30,667 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:30,667 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:30,667 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:30,670 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:30,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:30,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:30,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:30,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:30,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:30,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:30,671 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:30,674 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:30,674 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:30,674 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:30,674 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:30,674 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:30,674 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:30,674 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:30,674 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:30,675 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:30,675 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d690627297fb3f579eb1d6c5d71ead3a 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-05 04:57:30,675 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=A 2023-04-05 04:57:30,675 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:30,675 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=B 2023-04-05 04:57:30,675 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:30,675 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=C 2023-04-05 04:57:30,675 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:30,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on d690627297fb3f579eb1d6c5d71ead3a 2023-04-05 04:57:30,752 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:30,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:30,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41430 deadline: 1680670710780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:30,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:30,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1680670710854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:30,857 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:30,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41454 deadline: 1680670710856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:30,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:30,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41488 deadline: 1680670710862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:30,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:30,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41430 deadline: 1680670710885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:30,991 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:30,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41420 deadline: 1680670710990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:30,995 DEBUG [Thread-1618] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4154 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a., hostname=jenkins-hbase20.apache.org,38091,1680670569206, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 04:57:31,088 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:31,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41430 deadline: 1680670711087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:31,108 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=481 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/661365250f264bc78c1d6aed7a4049b9 2023-04-05 04:57:31,143 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=481 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/de20292049694a1082f605731f202c3d 2023-04-05 04:57:31,395 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:31,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41430 deadline: 1680670711392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:31,579 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=481 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/afdfbfb2f54a42c48181c95fa20a1ed7 2023-04-05 04:57:31,603 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/661365250f264bc78c1d6aed7a4049b9 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/661365250f264bc78c1d6aed7a4049b9 2023-04-05 04:57:31,608 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/661365250f264bc78c1d6aed7a4049b9, entries=150, sequenceid=481, filesize=11.9 K 2023-04-05 04:57:31,611 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/de20292049694a1082f605731f202c3d as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/de20292049694a1082f605731f202c3d 2023-04-05 04:57:31,659 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/de20292049694a1082f605731f202c3d, entries=150, sequenceid=481, filesize=11.9 K 2023-04-05 04:57:31,675 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/afdfbfb2f54a42c48181c95fa20a1ed7 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/afdfbfb2f54a42c48181c95fa20a1ed7 2023-04-05 04:57:31,679 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/afdfbfb2f54a42c48181c95fa20a1ed7, entries=150, sequenceid=481, filesize=11.9 K 2023-04-05 04:57:31,680 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for d690627297fb3f579eb1d6c5d71ead3a in 1005ms, sequenceid=481, compaction requested=true 2023-04-05 04:57:31,680 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:31,680 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:31,680 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:31,680 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:31,680 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:31,680 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:31,680 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:31,682 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:31,682 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:31,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:31,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:31,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:31,682 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:31,682 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:31,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:31,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:31,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:31,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:31,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:31,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:31,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a5c285[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:31,684 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:31,684 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:31,693 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:31,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:31,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:31,693 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:31,693 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:31,693 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:31,693 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:31,693 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:31,697 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:31,697 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:31,697 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:31,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:31,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:31,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:31,697 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:31,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:31,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:31,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:31,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:31,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:31,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:31,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:31,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:31,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:31,702 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:31,702 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:31,702 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:31,702 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:31,702 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:31,705 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:31,706 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:31,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:31,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:31,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:31,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:31,706 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:31,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:31,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:31,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:31,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:31,708 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:31,709 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:31,714 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:31,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:31,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:31,714 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:31,714 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:31,714 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:31,714 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:31,714 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:31,714 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:31,714 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:31,714 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:31,714 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:31,716 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:31,716 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:31,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:31,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:31,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:31,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:31,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:31,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:31,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:31,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:31,815 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:31,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:31,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:31,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:31,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:31,818 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:31,818 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:31,819 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:31,819 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:31,819 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:31,819 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:31,820 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:31,820 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:31,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:31,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:31,820 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:31,820 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:31,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:31,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:31,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:31,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:31,821 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:31,826 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:31,826 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:31,827 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:31,827 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:31,827 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:31,827 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:31,827 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:31,827 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:31,827 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:31,827 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d690627297fb3f579eb1d6c5d71ead3a 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-05 04:57:31,827 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=A 2023-04-05 04:57:31,827 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:31,828 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=B 2023-04-05 04:57:31,828 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:31,828 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=C 2023-04-05 04:57:31,828 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:31,848 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=493 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/63b3e61bd7364a43aa85718ec5a80e71 2023-04-05 04:57:31,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on d690627297fb3f579eb1d6c5d71ead3a 2023-04-05 04:57:31,864 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:31,892 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9554, totalHeapSize=21976, min timestamp=1680670651881, max timestamp=1680670651889Num uniques 859; 2023-04-05 04:57:31,892 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9558, totalHeapSize=21976, min timestamp=1680670651881, max timestamp=1680670651889Num uniques 859; 2023-04-05 04:57:31,892 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9554, totalHeapSize=21976, min timestamp=1680670651881, max timestamp=1680670651889Num uniques 859; 2023-04-05 04:57:31,892 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39324, new segment data size=9554, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:57:31,892 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39324, new segment data size=9554, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:57:31,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39328, new segment data size=9558, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:57:31,914 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:31,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1680670711910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:31,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:31,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41454 deadline: 1680670711912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:31,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:31,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41488 deadline: 1680670711915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:31,924 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:31,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41430 deadline: 1680670711921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:32,023 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:32,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41454 deadline: 1680670712020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:32,023 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:32,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1680670712021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:32,026 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:32,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41488 deadline: 1680670712022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:32,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:32,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41430 deadline: 1680670712028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:32,226 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:32,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1680670712225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:32,226 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:32,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41454 deadline: 1680670712225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:32,229 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:32,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41488 deadline: 1680670712227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:32,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:32,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41430 deadline: 1680670712234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:32,272 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=493 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/cb131b011cd844d79c27b227ddaac0c9 2023-04-05 04:57:32,295 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=493 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/1424a1aeab954868829d94cbcb7bf140 2023-04-05 04:57:32,303 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/63b3e61bd7364a43aa85718ec5a80e71 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/63b3e61bd7364a43aa85718ec5a80e71 2023-04-05 04:57:32,313 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/63b3e61bd7364a43aa85718ec5a80e71, entries=150, sequenceid=493, filesize=11.9 K 2023-04-05 04:57:32,314 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/cb131b011cd844d79c27b227ddaac0c9 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/cb131b011cd844d79c27b227ddaac0c9 2023-04-05 04:57:32,319 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/cb131b011cd844d79c27b227ddaac0c9, entries=150, sequenceid=493, filesize=11.9 K 2023-04-05 04:57:32,320 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/1424a1aeab954868829d94cbcb7bf140 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/1424a1aeab954868829d94cbcb7bf140 2023-04-05 04:57:32,328 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/1424a1aeab954868829d94cbcb7bf140, entries=150, sequenceid=493, filesize=11.9 K 2023-04-05 04:57:32,328 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for d690627297fb3f579eb1d6c5d71ead3a in 501ms, sequenceid=493, compaction requested=true 2023-04-05 04:57:32,328 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:32,328 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:32,329 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:32,329 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:32,329 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:32,329 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:32,329 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:32,330 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:32,330 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:32,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:32,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:32,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:32,331 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:32,331 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:32,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:32,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:32,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:32,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:32,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:32,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:32,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7afd3c1e[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:32,332 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:32,332 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:32,333 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:32,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:32,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:32,333 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:32,333 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:32,333 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:32,333 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:32,333 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:32,334 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:32,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:32,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:32,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:32,334 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:32,334 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:32,335 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:32,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:32,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:32,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:32,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:32,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:32,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:32,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:32,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:32,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:32,337 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:32,337 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:32,337 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:32,337 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:32,337 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:32,355 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:32,355 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:32,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:32,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:32,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:32,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:32,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:32,357 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:32,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:32,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:32,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:32,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:32,364 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:32,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:32,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:32,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:32,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:32,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:32,365 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:32,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:32,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:32,370 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:32,370 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:32,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:32,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:32,370 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:32,370 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:32,370 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:32,370 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:32,370 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:32,370 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:32,370 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:32,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:32,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:32,372 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:32,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:32,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:32,372 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:32,372 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:32,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:32,472 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:32,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:32,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:32,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:32,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:32,478 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:32,478 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:32,478 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:32,478 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:32,478 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:32,478 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:32,479 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:32,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:32,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:32,479 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:32,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:32,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:32,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:32,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:32,480 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:32,481 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:32,481 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:32,482 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:32,482 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:32,482 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:32,482 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:32,482 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:32,482 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:32,482 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:32,482 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:32,482 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:32,482 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d690627297fb3f579eb1d6c5d71ead3a 3/3 column families, dataSize=120.76 KB heapSize=340.45 KB 2023-04-05 04:57:32,483 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=A 2023-04-05 04:57:32,483 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:32,483 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=B 2023-04-05 04:57:32,483 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:32,483 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=C 2023-04-05 04:57:32,483 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:32,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on d690627297fb3f579eb1d6c5d71ead3a 2023-04-05 04:57:32,536 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:32,556 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:32,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41454 deadline: 1680670712550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:32,562 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:32,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1680670712556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:32,562 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:32,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41488 deadline: 1680670712556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:32,562 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:32,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41430 deadline: 1680670712558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:32,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:32,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41454 deadline: 1680670712658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:32,664 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:32,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1680670712663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:32,680 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:32,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41430 deadline: 1680670712677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:32,687 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:32,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41488 deadline: 1680670712683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:32,865 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:32,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41454 deadline: 1680670712862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:32,868 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:32,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1680670712866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:32,883 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:32,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41430 deadline: 1680670712882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:32,891 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:32,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41488 deadline: 1680670712888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:32,901 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=527 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/3570ef8e57c341b6b5369ea8f000b2f0 2023-04-05 04:57:32,936 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=527 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/d1c189c32c574324b3e15b41653a08ac 2023-04-05 04:57:32,972 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=527 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/251780e2b97a47deaa649813038d28fe 2023-04-05 04:57:32,985 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/3570ef8e57c341b6b5369ea8f000b2f0 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/3570ef8e57c341b6b5369ea8f000b2f0 2023-04-05 04:57:33,002 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/3570ef8e57c341b6b5369ea8f000b2f0, entries=150, sequenceid=527, filesize=11.9 K 2023-04-05 04:57:33,004 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/d1c189c32c574324b3e15b41653a08ac as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/d1c189c32c574324b3e15b41653a08ac 2023-04-05 04:57:33,009 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/d1c189c32c574324b3e15b41653a08ac, entries=150, sequenceid=527, filesize=11.9 K 2023-04-05 04:57:33,009 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/251780e2b97a47deaa649813038d28fe as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/251780e2b97a47deaa649813038d28fe 2023-04-05 04:57:33,018 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/251780e2b97a47deaa649813038d28fe, entries=150, sequenceid=527, filesize=11.9 K 2023-04-05 04:57:33,019 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.41 KB/348576, currentSize=67.09 KB/68700 for d690627297fb3f579eb1d6c5d71ead3a in 537ms, sequenceid=527, compaction requested=true 2023-04-05 04:57:33,019 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:33,019 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:33,019 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:33,019 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:33,019 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:33,019 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:33,020 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:33,020 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:33,020 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:33,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:33,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:33,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:33,021 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:33,021 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:33,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:33,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:33,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:33,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:33,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:33,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:33,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f2fdf7b[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:33,022 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:33,022 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:33,022 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:33,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:33,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:33,023 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:33,023 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:33,023 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:33,023 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:33,023 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:33,023 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:33,023 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:33,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:33,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:33,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:33,023 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:33,024 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:33,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:33,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:33,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:33,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:33,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:33,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:33,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:33,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:33,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:33,025 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:33,026 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:33,026 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:33,026 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:33,026 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:33,026 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:33,027 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:33,027 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:33,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:33,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:33,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:33,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:33,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:33,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:33,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:33,027 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:33,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:33,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:33,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:33,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:33,028 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:33,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:33,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:33,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:33,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:33,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:33,031 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:33,031 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:33,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:33,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:33,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:33,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:33,037 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:33,037 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:33,037 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:33,037 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:33,037 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:33,037 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:33,037 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:33,037 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:33,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:33,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:33,037 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:33,037 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:33,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:33,137 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:33,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:33,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:33,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:33,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:33,154 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:33,154 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:33,154 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:33,154 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:33,156 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:33,156 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:33,166 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:33,166 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:33,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:33,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:33,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:33,166 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:33,166 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:33,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:33,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:33,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:33,167 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:33,169 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:33,169 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:33,169 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:33,169 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:33,169 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:33,170 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:33,170 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:33,170 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:33,170 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:33,170 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d690627297fb3f579eb1d6c5d71ead3a 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-05 04:57:33,170 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=A 2023-04-05 04:57:33,170 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:33,171 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=B 2023-04-05 04:57:33,171 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:33,171 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=C 2023-04-05 04:57:33,171 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:33,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on d690627297fb3f579eb1d6c5d71ead3a 2023-04-05 04:57:33,174 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:33,221 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:33,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1680670713214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:33,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:33,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41454 deadline: 1680670713218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:33,224 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:33,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41488 deadline: 1680670713218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:33,224 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:33,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41430 deadline: 1680670713218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:33,326 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:33,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1680670713322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:33,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:33,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41488 deadline: 1680670713326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:33,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:33,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41430 deadline: 1680670713326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:33,332 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:33,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41454 deadline: 1680670713328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:33,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:33,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41488 deadline: 1680670713532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:33,535 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:33,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41430 deadline: 1680670713532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:33,539 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:33,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41454 deadline: 1680670713536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:33,540 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:33,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1680670713538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:33,586 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=540 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/a4b80b544be5451cafaf33e37283d1e3 2023-04-05 04:57:33,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:33,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41488 deadline: 1680670713836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:33,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:33,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41430 deadline: 1680670713837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:33,846 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:33,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41454 deadline: 1680670713844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:33,847 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:33,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1680670713844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:34,013 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=540 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/d5f059abe23d4202b224400851348756 2023-04-05 04:57:34,344 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:34,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41488 deadline: 1680670714343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:34,347 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:34,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41430 deadline: 1680670714344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:34,351 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:34,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41454 deadline: 1680670714348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:34,352 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:34,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1680670714350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:34,441 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=540 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/774eec17d1974c8cbe17caeb4a254c7d 2023-04-05 04:57:34,451 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/a4b80b544be5451cafaf33e37283d1e3 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/a4b80b544be5451cafaf33e37283d1e3 2023-04-05 04:57:34,456 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/a4b80b544be5451cafaf33e37283d1e3, entries=150, sequenceid=540, filesize=11.9 K 2023-04-05 04:57:34,456 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/d5f059abe23d4202b224400851348756 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/d5f059abe23d4202b224400851348756 2023-04-05 04:57:34,466 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/d5f059abe23d4202b224400851348756, entries=150, sequenceid=540, filesize=11.9 K 2023-04-05 04:57:34,467 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/774eec17d1974c8cbe17caeb4a254c7d as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/774eec17d1974c8cbe17caeb4a254c7d 2023-04-05 04:57:34,482 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/774eec17d1974c8cbe17caeb4a254c7d, entries=150, sequenceid=540, filesize=11.9 K 2023-04-05 04:57:34,483 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for d690627297fb3f579eb1d6c5d71ead3a in 1313ms, sequenceid=540, compaction requested=true 2023-04-05 04:57:34,483 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:34,483 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:34,483 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:34,483 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:34,483 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:34,483 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:34,483 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:34,484 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:34,484 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:34,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:34,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:34,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:34,485 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:34,485 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:34,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:34,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:34,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:34,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:34,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:34,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:34,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@771755fa[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:34,486 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:34,486 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:34,489 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:34,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:34,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:34,489 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:34,489 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:34,489 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:34,496 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:34,496 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:34,502 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:34,502 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:34,502 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:34,502 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:34,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:34,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:34,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:34,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:34,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:34,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:34,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:34,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:34,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:34,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:34,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:34,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:34,504 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:34,505 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:34,505 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:34,505 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:34,505 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:34,507 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:34,507 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:34,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:34,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:34,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:34,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:34,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:34,507 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:34,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:34,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:34,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:34,509 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:34,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:34,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:34,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:34,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:34,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:34,512 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:34,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:34,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:34,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:34,518 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:34,519 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:34,519 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:34,526 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:34,526 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:34,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:34,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:34,526 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:34,526 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:34,527 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:34,527 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:34,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:34,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:34,530 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:34,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:34,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:34,530 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:34,530 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:34,630 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:34,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:34,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:34,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:34,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:34,639 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:34,639 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:34,639 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:34,639 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:34,642 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:34,642 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:34,643 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:34,643 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:34,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:34,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:34,643 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:34,643 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:34,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:34,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:34,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:34,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:34,644 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:34,650 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:34,650 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:34,650 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:34,650 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:34,650 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:34,655 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:34,656 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:34,662 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:34,662 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. started... 2023-04-05 04:57:34,662 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d690627297fb3f579eb1d6c5d71ead3a 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-05 04:57:34,662 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=A 2023-04-05 04:57:34,662 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:34,662 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=B 2023-04-05 04:57:34,662 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:34,662 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=C 2023-04-05 04:57:34,662 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:35,015 DEBUG [Thread-1623] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7bae1a64 to 127.0.0.1:56229 2023-04-05 04:57:35,015 DEBUG [Thread-1623] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:57:35,017 DEBUG [Thread-1625] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5654f8ca to 127.0.0.1:56229 2023-04-05 04:57:35,017 DEBUG [Thread-1625] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:57:35,018 DEBUG [Thread-1627] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x509adaa8 to 127.0.0.1:56229 2023-04-05 04:57:35,018 DEBUG [Thread-1627] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:57:35,020 DEBUG [Thread-1621] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6b458e37 to 127.0.0.1:56229 2023-04-05 04:57:35,020 DEBUG [Thread-1621] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:57:35,021 DEBUG [Thread-1629] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e5e7a93 to 127.0.0.1:56229 2023-04-05 04:57:35,021 DEBUG [Thread-1629] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:57:35,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on d690627297fb3f579eb1d6c5d71ead3a 2023-04-05 04:57:35,031 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. as already flushing 2023-04-05 04:57:35,032 DEBUG [Thread-1618] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x62be7a0b to 127.0.0.1:56229 2023-04-05 04:57:35,032 DEBUG [Thread-1618] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:57:35,103 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=559 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/13dbe07bb4284ef4ac4f9e2dde65fe09 2023-04-05 04:57:35,349 DEBUG [Thread-1612] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x712d78ca to 127.0.0.1:56229 2023-04-05 04:57:35,349 DEBUG [Thread-1612] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:57:35,356 DEBUG [Thread-1614] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x407b9cb0 to 127.0.0.1:56229 2023-04-05 04:57:35,356 DEBUG [Thread-1614] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:57:35,359 DEBUG [Thread-1610] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7187fcf3 to 127.0.0.1:56229 2023-04-05 04:57:35,359 DEBUG [Thread-1610] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:57:35,360 DEBUG [Thread-1616] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7f456fa2 to 127.0.0.1:56229 2023-04-05 04:57:35,360 DEBUG [Thread-1616] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:57:35,515 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=559 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/8c15d3d8b3dd4b3e87ac2fe2391eb93a 2023-04-05 04:57:35,928 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=559 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/77723c66a07b4382a0f80c9c3b124501 2023-04-05 04:57:35,932 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/13dbe07bb4284ef4ac4f9e2dde65fe09 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/13dbe07bb4284ef4ac4f9e2dde65fe09 2023-04-05 04:57:35,936 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/13dbe07bb4284ef4ac4f9e2dde65fe09, entries=150, sequenceid=559, filesize=11.9 K 2023-04-05 04:57:35,936 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/8c15d3d8b3dd4b3e87ac2fe2391eb93a as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/8c15d3d8b3dd4b3e87ac2fe2391eb93a 2023-04-05 04:57:35,940 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/8c15d3d8b3dd4b3e87ac2fe2391eb93a, entries=150, sequenceid=559, filesize=11.9 K 2023-04-05 04:57:35,941 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/77723c66a07b4382a0f80c9c3b124501 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/77723c66a07b4382a0f80c9c3b124501 2023-04-05 04:57:35,944 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/77723c66a07b4382a0f80c9c3b124501, entries=150, sequenceid=559, filesize=11.9 K 2023-04-05 04:57:35,945 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=33.54 KB/34350 for d690627297fb3f579eb1d6c5d71ead3a in 1283ms, sequenceid=559, compaction requested=true 2023-04-05 04:57:35,945 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:35,945 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:35,945 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:35,945 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:35,945 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:35,945 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:35,945 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:35,946 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:35,946 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:35,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:35,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:35,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:35,946 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:35,946 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:35,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:35,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:35,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:35,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:35,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:35,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:35,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a8b1077[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:35,947 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:35,947 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:35,948 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:35,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:35,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:35,948 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:35,948 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:35,948 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:35,948 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:35,948 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:35,949 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:35,949 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:35,949 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:35,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:35,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:35,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:35,949 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:35,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:35,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:35,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:35,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:35,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:35,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:35,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:35,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:35,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:35,950 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:35,951 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:35,951 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:35,951 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:35,951 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:35,951 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:35,951 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:35,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:35,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:35,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:35,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:35,951 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:35,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:35,952 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:35,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:35,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:35,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:35,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:35,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:35,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:35,952 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:35,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:35,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:35,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:35,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:35,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:35,954 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:35,954 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:35,954 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:35,954 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:35,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:35,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:35,954 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:35,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:35,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:35,954 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:35,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:35,954 DEBUG [Thread-1620] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:35,954 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:35,954 DEBUG [Thread-1620] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:35,954 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:35,954 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:35,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:35,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:35,954 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:36,054 DEBUG [Thread-1620] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:36,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:36,055 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-05 04:57:36,056 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(394): wrote 91 2023-04-05 04:57:36,056 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(394): wrote 76 2023-04-05 04:57:36,056 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(394): wrote 86 2023-04-05 04:57:36,056 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(394): wrote 79 2023-04-05 04:57:36,056 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(394): wrote 138 2023-04-05 04:57:36,056 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-05 04:57:36,056 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(398): read 4526 2023-04-05 04:57:36,056 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(398): read 4623 2023-04-05 04:57:36,056 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(398): read 4312 2023-04-05 04:57:36,056 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(398): read 4511 2023-04-05 04:57:36,056 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(398): read 4666 2023-04-05 04:57:36,056 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-05 04:57:36,056 INFO [Listener at localhost.localdomain/34579] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-05 04:57:36,056 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4d3f4b11 to 127.0.0.1:56229 2023-04-05 04:57:36,056 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:57:36,057 INFO [Listener at localhost.localdomain/34579] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-05 04:57:36,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-05 04:57:36,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-05 04:57:36,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-05 04:57:36,060 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680670656060"}]},"ts":"1680670656060"} 2023-04-05 04:57:36,061 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-05 04:57:36,062 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-05 04:57:36,063 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d690627297fb3f579eb1d6c5d71ead3a, UNASSIGN}] 2023-04-05 04:57:36,064 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d690627297fb3f579eb1d6c5d71ead3a, UNASSIGN 2023-04-05 04:57:36,065 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=d690627297fb3f579eb1d6c5d71ead3a, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:36,065 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670656065"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680670656065"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680670656065"}]},"ts":"1680670656065"} 2023-04-05 04:57:36,066 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206}] 2023-04-05 04:57:36,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-05 04:57:36,218 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close d690627297fb3f579eb1d6c5d71ead3a 2023-04-05 04:57:36,219 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing d690627297fb3f579eb1d6c5d71ead3a, disabling compactions & flushes 2023-04-05 04:57:36,219 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:36,219 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:36,219 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. after waiting 0 ms 2023-04-05 04:57:36,219 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:36,219 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing d690627297fb3f579eb1d6c5d71ead3a 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-05 04:57:36,219 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=A 2023-04-05 04:57:36,220 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:36,220 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=B 2023-04-05 04:57:36,220 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:36,220 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d690627297fb3f579eb1d6c5d71ead3a, store=C 2023-04-05 04:57:36,220 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:36,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-05 04:57:36,632 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=567 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/95fc4fa411c840d5b6f483d2e04a25dc 2023-04-05 04:57:36,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-05 04:57:37,052 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=567 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/22c4fdcfdb48434086b6ec2ff9d8d267 2023-04-05 04:57:37,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-05 04:57:37,470 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=567 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/dcdb1caf19874686b610e1ebcd56b049 2023-04-05 04:57:37,482 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/A/95fc4fa411c840d5b6f483d2e04a25dc as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/95fc4fa411c840d5b6f483d2e04a25dc 2023-04-05 04:57:37,487 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/95fc4fa411c840d5b6f483d2e04a25dc, entries=150, sequenceid=567, filesize=11.9 K 2023-04-05 04:57:37,487 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/B/22c4fdcfdb48434086b6ec2ff9d8d267 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/22c4fdcfdb48434086b6ec2ff9d8d267 2023-04-05 04:57:37,491 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/22c4fdcfdb48434086b6ec2ff9d8d267, entries=150, sequenceid=567, filesize=11.9 K 2023-04-05 04:57:37,492 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/.tmp/C/dcdb1caf19874686b610e1ebcd56b049 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/dcdb1caf19874686b610e1ebcd56b049 2023-04-05 04:57:37,495 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/dcdb1caf19874686b610e1ebcd56b049, entries=150, sequenceid=567, filesize=11.9 K 2023-04-05 04:57:37,496 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for d690627297fb3f579eb1d6c5d71ead3a in 1277ms, sequenceid=567, compaction requested=true 2023-04-05 04:57:37,516 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/e03765a3b5fa4ab7a2934d24ba02a64e, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/e3a9273c3384408ba281d860bfbc81dd, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/5898d13b365c42379357b107a35c0ae5, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/71b1752120ff4768b9da68b665bfdf0c, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/0fe53263e7e64c38aa4c8b363f1d4d89, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/9923316413824b0e807ef015f987c70f, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/7482240a5d7248229e208d1009e062d7, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/8258a49745c34790a345b242f5d591da, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/fb3d74f18d3d4d43877076ba725e6186, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/87227a068d3d4b838c82cbba2437c6ec, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/a98067121af740638dfbec3d0caf259c, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/77adc92b05294903a0f6e1ffc02de278, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/d2ae4f3caae34c5c9512206d89efd486, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/adec1d8788274ffb908c2021ec4fb471, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/b30329d40e2349a48fe9dffd5a0a963d, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/9b12e5385102444a978e2d586bc00dca, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/be5f13bb192c4555b0627c2187c25102, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/6700a2ca8c8744d2b01a6baae69c046b, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/6a80bc29688345d1b7962755b65f3fb3, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/cb3b7540a979429bad8205005206675a, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/3a2201b280694da7aff1214308b02e33, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/a0b8aacb12dc4336bb024ac62285a3d8, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/777c55cebd764a97b86a5985234c14ad] to archive 2023-04-05 04:57:37,516 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 04:57:37,518 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/e03765a3b5fa4ab7a2934d24ba02a64e to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/e03765a3b5fa4ab7a2934d24ba02a64e 2023-04-05 04:57:37,519 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/e3a9273c3384408ba281d860bfbc81dd to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/e3a9273c3384408ba281d860bfbc81dd 2023-04-05 04:57:37,520 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/5898d13b365c42379357b107a35c0ae5 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/5898d13b365c42379357b107a35c0ae5 2023-04-05 04:57:37,521 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/71b1752120ff4768b9da68b665bfdf0c to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/71b1752120ff4768b9da68b665bfdf0c 2023-04-05 04:57:37,522 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/0fe53263e7e64c38aa4c8b363f1d4d89 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/0fe53263e7e64c38aa4c8b363f1d4d89 2023-04-05 04:57:37,522 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/9923316413824b0e807ef015f987c70f to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/9923316413824b0e807ef015f987c70f 2023-04-05 04:57:37,523 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/7482240a5d7248229e208d1009e062d7 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/7482240a5d7248229e208d1009e062d7 2023-04-05 04:57:37,524 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/8258a49745c34790a345b242f5d591da to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/8258a49745c34790a345b242f5d591da 2023-04-05 04:57:37,525 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/fb3d74f18d3d4d43877076ba725e6186 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/fb3d74f18d3d4d43877076ba725e6186 2023-04-05 04:57:37,526 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/87227a068d3d4b838c82cbba2437c6ec to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/87227a068d3d4b838c82cbba2437c6ec 2023-04-05 04:57:37,527 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/a98067121af740638dfbec3d0caf259c to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/a98067121af740638dfbec3d0caf259c 2023-04-05 04:57:37,527 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/77adc92b05294903a0f6e1ffc02de278 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/77adc92b05294903a0f6e1ffc02de278 2023-04-05 04:57:37,528 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/d2ae4f3caae34c5c9512206d89efd486 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/d2ae4f3caae34c5c9512206d89efd486 2023-04-05 04:57:37,529 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/adec1d8788274ffb908c2021ec4fb471 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/adec1d8788274ffb908c2021ec4fb471 2023-04-05 04:57:37,529 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/b30329d40e2349a48fe9dffd5a0a963d to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/b30329d40e2349a48fe9dffd5a0a963d 2023-04-05 04:57:37,530 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/9b12e5385102444a978e2d586bc00dca to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/9b12e5385102444a978e2d586bc00dca 2023-04-05 04:57:37,531 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/be5f13bb192c4555b0627c2187c25102 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/be5f13bb192c4555b0627c2187c25102 2023-04-05 04:57:37,532 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/6700a2ca8c8744d2b01a6baae69c046b to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/6700a2ca8c8744d2b01a6baae69c046b 2023-04-05 04:57:37,533 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/6a80bc29688345d1b7962755b65f3fb3 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/6a80bc29688345d1b7962755b65f3fb3 2023-04-05 04:57:37,534 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/cb3b7540a979429bad8205005206675a to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/cb3b7540a979429bad8205005206675a 2023-04-05 04:57:37,535 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/3a2201b280694da7aff1214308b02e33 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/3a2201b280694da7aff1214308b02e33 2023-04-05 04:57:37,536 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/a0b8aacb12dc4336bb024ac62285a3d8 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/a0b8aacb12dc4336bb024ac62285a3d8 2023-04-05 04:57:37,537 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/777c55cebd764a97b86a5985234c14ad to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/777c55cebd764a97b86a5985234c14ad 2023-04-05 04:57:37,541 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/f93c90844a7f4b52933ee0214ed0a06f, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/13fefedd2c76485bbf3b3b484d1c81c0, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/95e6c856f88345e6b41243056b76d376, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/baf1242aeb6a488ebcd64227638117b1, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/0dce95b61fca4325863fbb8812756c3f, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/3d1a803bb9da40d09b223da411e1e9db, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/43b3a9e64b594fd2a3232f1416f0b992, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/066c254ebe9f40e1972994af802ef689, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/f3ba9e4bb54a47fe9f0f1580493ba681, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/a80eea887a94448996db0e8ab4cbf41c, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/6b8f353e24594de498f671461c898f51, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/e34048afa1c14c8e8104113a553c894e, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/f3a0d93bfaa8444eae4a6d4376251b62, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/c47359947b914dd89fc762524a8ab105, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/d1af6161321740ec9f5e55609266fc36, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/76fad2de8c5b43a8a6050baa5ce78411, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/e02f4d150763452fbd217ec62f52bdb9, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/a57987ebd5384b15a1a327ff0dc87907, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/0a1cfc48f53f46bd8d1489a3a4b22490, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/8335910e42c0404e87ce76f7146447bd, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/40004b38dadd4b00a5a2afed685b8e51, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/edc1a6a7aaab4f06b68076de9c507a2f, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/ab5097a1a60d4ce69e0aa02d8c54776a] to archive 2023-04-05 04:57:37,541 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 04:57:37,543 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/f93c90844a7f4b52933ee0214ed0a06f to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/f93c90844a7f4b52933ee0214ed0a06f 2023-04-05 04:57:37,544 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/13fefedd2c76485bbf3b3b484d1c81c0 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/13fefedd2c76485bbf3b3b484d1c81c0 2023-04-05 04:57:37,545 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/95e6c856f88345e6b41243056b76d376 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/95e6c856f88345e6b41243056b76d376 2023-04-05 04:57:37,546 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/baf1242aeb6a488ebcd64227638117b1 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/baf1242aeb6a488ebcd64227638117b1 2023-04-05 04:57:37,547 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/0dce95b61fca4325863fbb8812756c3f to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/0dce95b61fca4325863fbb8812756c3f 2023-04-05 04:57:37,548 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/3d1a803bb9da40d09b223da411e1e9db to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/3d1a803bb9da40d09b223da411e1e9db 2023-04-05 04:57:37,549 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/43b3a9e64b594fd2a3232f1416f0b992 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/43b3a9e64b594fd2a3232f1416f0b992 2023-04-05 04:57:37,550 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/066c254ebe9f40e1972994af802ef689 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/066c254ebe9f40e1972994af802ef689 2023-04-05 04:57:37,551 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/f3ba9e4bb54a47fe9f0f1580493ba681 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/f3ba9e4bb54a47fe9f0f1580493ba681 2023-04-05 04:57:37,552 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/a80eea887a94448996db0e8ab4cbf41c to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/a80eea887a94448996db0e8ab4cbf41c 2023-04-05 04:57:37,553 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/6b8f353e24594de498f671461c898f51 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/6b8f353e24594de498f671461c898f51 2023-04-05 04:57:37,554 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/e34048afa1c14c8e8104113a553c894e to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/e34048afa1c14c8e8104113a553c894e 2023-04-05 04:57:37,555 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/f3a0d93bfaa8444eae4a6d4376251b62 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/f3a0d93bfaa8444eae4a6d4376251b62 2023-04-05 04:57:37,557 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/c47359947b914dd89fc762524a8ab105 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/c47359947b914dd89fc762524a8ab105 2023-04-05 04:57:37,558 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/d1af6161321740ec9f5e55609266fc36 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/d1af6161321740ec9f5e55609266fc36 2023-04-05 04:57:37,559 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/76fad2de8c5b43a8a6050baa5ce78411 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/76fad2de8c5b43a8a6050baa5ce78411 2023-04-05 04:57:37,560 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/e02f4d150763452fbd217ec62f52bdb9 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/e02f4d150763452fbd217ec62f52bdb9 2023-04-05 04:57:37,561 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/a57987ebd5384b15a1a327ff0dc87907 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/a57987ebd5384b15a1a327ff0dc87907 2023-04-05 04:57:37,563 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/0a1cfc48f53f46bd8d1489a3a4b22490 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/0a1cfc48f53f46bd8d1489a3a4b22490 2023-04-05 04:57:37,564 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/8335910e42c0404e87ce76f7146447bd to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/8335910e42c0404e87ce76f7146447bd 2023-04-05 04:57:37,565 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/40004b38dadd4b00a5a2afed685b8e51 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/40004b38dadd4b00a5a2afed685b8e51 2023-04-05 04:57:37,566 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/edc1a6a7aaab4f06b68076de9c507a2f to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/edc1a6a7aaab4f06b68076de9c507a2f 2023-04-05 04:57:37,568 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/ab5097a1a60d4ce69e0aa02d8c54776a to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/ab5097a1a60d4ce69e0aa02d8c54776a 2023-04-05 04:57:37,572 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/a7eacc04f5204c82977f172a667413d5, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/e8711c202c63409987a5888a3935781d, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/fdcdce7652c84cd2ad63738fd66c27bf, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/064dce90bec24c8e887fb7a138f164ed, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/4902aed2c06a4511864e5c1ed071ee78, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/526986ff88b042df82bc67adb04a27d9, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/764ba28c608f4718895e8cf8217945bd, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/ed96b7557619406b8fe50d34c9acd7d5, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/52d9161460d64d6781300459898d67ed, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/f2e3a4aa8b7b46f78af865ef6eb4fb8f, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/a19bafbf862e4c5baf27c8ab51032768, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/7c17be52b9b441bfb50c9b2b023c76df, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/42ab4f5e223d4d1ca3ef2f12b4ce83dc, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/2afefd0f86134e25996c9823c5b75325, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/8d6fb8800aa24e4996d4b52ff0269775, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/f5c062b08845408cab8418b68105784c, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/c5b77dbfc23648589ad4bfe582894d85, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/69c6760c7d2a48dbab37dd28ddb3c4b5, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/32f0fbeede894e8a94cd35090cf0200b, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/c084d15e57814736bf93bc25f4d75b48, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/ffc7cb5d86094100a3e96f39c713d0c0, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/50c03d0a649b4f699bef2c465c21b5e3, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/5cb7557e1da8445db174319da25ad558] to archive 2023-04-05 04:57:37,573 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 04:57:37,575 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/a7eacc04f5204c82977f172a667413d5 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/a7eacc04f5204c82977f172a667413d5 2023-04-05 04:57:37,576 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/e8711c202c63409987a5888a3935781d to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/e8711c202c63409987a5888a3935781d 2023-04-05 04:57:37,577 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/fdcdce7652c84cd2ad63738fd66c27bf to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/fdcdce7652c84cd2ad63738fd66c27bf 2023-04-05 04:57:37,577 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/064dce90bec24c8e887fb7a138f164ed to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/064dce90bec24c8e887fb7a138f164ed 2023-04-05 04:57:37,578 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/4902aed2c06a4511864e5c1ed071ee78 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/4902aed2c06a4511864e5c1ed071ee78 2023-04-05 04:57:37,579 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/526986ff88b042df82bc67adb04a27d9 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/526986ff88b042df82bc67adb04a27d9 2023-04-05 04:57:37,580 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/764ba28c608f4718895e8cf8217945bd to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/764ba28c608f4718895e8cf8217945bd 2023-04-05 04:57:37,581 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/ed96b7557619406b8fe50d34c9acd7d5 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/ed96b7557619406b8fe50d34c9acd7d5 2023-04-05 04:57:37,581 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/52d9161460d64d6781300459898d67ed to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/52d9161460d64d6781300459898d67ed 2023-04-05 04:57:37,582 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/f2e3a4aa8b7b46f78af865ef6eb4fb8f to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/f2e3a4aa8b7b46f78af865ef6eb4fb8f 2023-04-05 04:57:37,583 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/a19bafbf862e4c5baf27c8ab51032768 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/a19bafbf862e4c5baf27c8ab51032768 2023-04-05 04:57:37,584 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/7c17be52b9b441bfb50c9b2b023c76df to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/7c17be52b9b441bfb50c9b2b023c76df 2023-04-05 04:57:37,584 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/42ab4f5e223d4d1ca3ef2f12b4ce83dc to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/42ab4f5e223d4d1ca3ef2f12b4ce83dc 2023-04-05 04:57:37,585 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/2afefd0f86134e25996c9823c5b75325 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/2afefd0f86134e25996c9823c5b75325 2023-04-05 04:57:37,586 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/8d6fb8800aa24e4996d4b52ff0269775 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/8d6fb8800aa24e4996d4b52ff0269775 2023-04-05 04:57:37,587 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/f5c062b08845408cab8418b68105784c to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/f5c062b08845408cab8418b68105784c 2023-04-05 04:57:37,587 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/c5b77dbfc23648589ad4bfe582894d85 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/c5b77dbfc23648589ad4bfe582894d85 2023-04-05 04:57:37,588 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/69c6760c7d2a48dbab37dd28ddb3c4b5 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/69c6760c7d2a48dbab37dd28ddb3c4b5 2023-04-05 04:57:37,589 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/32f0fbeede894e8a94cd35090cf0200b to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/32f0fbeede894e8a94cd35090cf0200b 2023-04-05 04:57:37,590 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/c084d15e57814736bf93bc25f4d75b48 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/c084d15e57814736bf93bc25f4d75b48 2023-04-05 04:57:37,591 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/ffc7cb5d86094100a3e96f39c713d0c0 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/ffc7cb5d86094100a3e96f39c713d0c0 2023-04-05 04:57:37,591 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/50c03d0a649b4f699bef2c465c21b5e3 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/50c03d0a649b4f699bef2c465c21b5e3 2023-04-05 04:57:37,592 DEBUG [StoreCloser-TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/5cb7557e1da8445db174319da25ad558 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/5cb7557e1da8445db174319da25ad558 2023-04-05 04:57:37,596 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/recovered.edits/570.seqid, newMaxSeqId=570, maxSeqId=1 2023-04-05 04:57:37,597 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a. 2023-04-05 04:57:37,597 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for d690627297fb3f579eb1d6c5d71ead3a: 2023-04-05 04:57:37,598 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed d690627297fb3f579eb1d6c5d71ead3a 2023-04-05 04:57:37,598 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=d690627297fb3f579eb1d6c5d71ead3a, regionState=CLOSED 2023-04-05 04:57:37,599 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670657598"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680670657598"}]},"ts":"1680670657598"} 2023-04-05 04:57:37,601 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-05 04:57:37,601 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure d690627297fb3f579eb1d6c5d71ead3a, server=jenkins-hbase20.apache.org,38091,1680670569206 in 1.5330 sec 2023-04-05 04:57:37,602 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-05 04:57:37,602 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=d690627297fb3f579eb1d6c5d71ead3a, UNASSIGN in 1.5380 sec 2023-04-05 04:57:37,602 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680670657602"}]},"ts":"1680670657602"} 2023-04-05 04:57:37,603 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-05 04:57:37,604 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-05 04:57:37,605 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5470 sec 2023-04-05 04:57:38,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-05 04:57:38,171 INFO [Listener at localhost.localdomain/34579] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-05 04:57:38,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-05 04:57:38,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 04:57:38,179 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 04:57:38,180 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 04:57:38,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-05 04:57:38,184 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a 2023-04-05 04:57:38,186 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A, FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B, FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C, FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/recovered.edits] 2023-04-05 04:57:38,188 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/13dbe07bb4284ef4ac4f9e2dde65fe09 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/13dbe07bb4284ef4ac4f9e2dde65fe09 2023-04-05 04:57:38,189 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/3570ef8e57c341b6b5369ea8f000b2f0 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/3570ef8e57c341b6b5369ea8f000b2f0 2023-04-05 04:57:38,190 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/63b3e61bd7364a43aa85718ec5a80e71 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/63b3e61bd7364a43aa85718ec5a80e71 2023-04-05 04:57:38,191 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/661365250f264bc78c1d6aed7a4049b9 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/661365250f264bc78c1d6aed7a4049b9 2023-04-05 04:57:38,192 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/95fc4fa411c840d5b6f483d2e04a25dc to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/95fc4fa411c840d5b6f483d2e04a25dc 2023-04-05 04:57:38,193 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/988569b968af41f7bdbc5bc538f7d9e1 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/988569b968af41f7bdbc5bc538f7d9e1 2023-04-05 04:57:38,194 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/a4b80b544be5451cafaf33e37283d1e3 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/a4b80b544be5451cafaf33e37283d1e3 2023-04-05 04:57:38,196 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/c43833756b72428a88662f6b34bc3ed9 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/c43833756b72428a88662f6b34bc3ed9 2023-04-05 04:57:38,197 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/fe2edbde9dd84addb90eceb2909ad773 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/A/fe2edbde9dd84addb90eceb2909ad773 2023-04-05 04:57:38,199 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/1a87b4e4ebcc476c9ee046301edeb0b9 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/1a87b4e4ebcc476c9ee046301edeb0b9 2023-04-05 04:57:38,200 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/22c4fdcfdb48434086b6ec2ff9d8d267 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/22c4fdcfdb48434086b6ec2ff9d8d267 2023-04-05 04:57:38,201 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/8c15d3d8b3dd4b3e87ac2fe2391eb93a to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/8c15d3d8b3dd4b3e87ac2fe2391eb93a 2023-04-05 04:57:38,203 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/cb131b011cd844d79c27b227ddaac0c9 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/cb131b011cd844d79c27b227ddaac0c9 2023-04-05 04:57:38,204 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/cde7374cfdfa48238b9c96075fa15596 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/cde7374cfdfa48238b9c96075fa15596 2023-04-05 04:57:38,205 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/d1c189c32c574324b3e15b41653a08ac to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/d1c189c32c574324b3e15b41653a08ac 2023-04-05 04:57:38,206 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/d5f059abe23d4202b224400851348756 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/d5f059abe23d4202b224400851348756 2023-04-05 04:57:38,207 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/dac6339a6350410bb7e0019e0df2ecb0 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/dac6339a6350410bb7e0019e0df2ecb0 2023-04-05 04:57:38,209 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/de20292049694a1082f605731f202c3d to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/B/de20292049694a1082f605731f202c3d 2023-04-05 04:57:38,211 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/1424a1aeab954868829d94cbcb7bf140 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/1424a1aeab954868829d94cbcb7bf140 2023-04-05 04:57:38,213 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/251780e2b97a47deaa649813038d28fe to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/251780e2b97a47deaa649813038d28fe 2023-04-05 04:57:38,215 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/4a9eec3c96b14290b7d4c5fe71a83cc1 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/4a9eec3c96b14290b7d4c5fe71a83cc1 2023-04-05 04:57:38,217 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/774eec17d1974c8cbe17caeb4a254c7d to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/774eec17d1974c8cbe17caeb4a254c7d 2023-04-05 04:57:38,219 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/77723c66a07b4382a0f80c9c3b124501 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/77723c66a07b4382a0f80c9c3b124501 2023-04-05 04:57:38,221 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/9aeed6a50c6d496b8505720df74a72e3 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/9aeed6a50c6d496b8505720df74a72e3 2023-04-05 04:57:38,223 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/afdfbfb2f54a42c48181c95fa20a1ed7 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/afdfbfb2f54a42c48181c95fa20a1ed7 2023-04-05 04:57:38,225 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/dcdb1caf19874686b610e1ebcd56b049 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/dcdb1caf19874686b610e1ebcd56b049 2023-04-05 04:57:38,226 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/e7360a41ceeb4747a6aa2121ba1fb83d to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/C/e7360a41ceeb4747a6aa2121ba1fb83d 2023-04-05 04:57:38,230 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/recovered.edits/570.seqid to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a/recovered.edits/570.seqid 2023-04-05 04:57:38,231 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/d690627297fb3f579eb1d6c5d71ead3a 2023-04-05 04:57:38,231 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-05 04:57:38,234 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 04:57:38,236 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-05 04:57:38,238 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-05 04:57:38,239 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 04:57:38,239 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-05 04:57:38,239 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680670658239"}]},"ts":"9223372036854775807"} 2023-04-05 04:57:38,242 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-05 04:57:38,242 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => d690627297fb3f579eb1d6c5d71ead3a, NAME => 'TestAcidGuarantees,,1680670632804.d690627297fb3f579eb1d6c5d71ead3a.', STARTKEY => '', ENDKEY => ''}] 2023-04-05 04:57:38,242 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-05 04:57:38,242 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680670658242"}]},"ts":"9223372036854775807"} 2023-04-05 04:57:38,244 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-05 04:57:38,246 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 04:57:38,247 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 73 msec 2023-04-05 04:57:38,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-05 04:57:38,282 INFO [Listener at localhost.localdomain/34579] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-05 04:57:38,302 INFO [Listener at localhost.localdomain/34579] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=228 (was 227) - Thread LEAK? -, OpenFileDescriptor=539 (was 509) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=482 (was 287) - SystemLoadAverage LEAK? -, ProcessCount=166 (was 166), AvailableMemoryMB=6337 (was 6384) 2023-04-05 04:57:38,314 INFO [Listener at localhost.localdomain/34579] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=228, OpenFileDescriptor=539, MaxFileDescriptor=60000, SystemLoadAverage=482, ProcessCount=166, AvailableMemoryMB=6336 2023-04-05 04:57:38,315 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-05 04:57:38,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-05 04:57:38,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-05 04:57:38,318 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-05 04:57:38,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-05 04:57:38,319 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-05 04:57:38,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-05 04:57:38,320 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:57:38,320 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0 empty. 2023-04-05 04:57:38,320 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:57:38,321 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-05 04:57:38,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-05 04:57:38,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-05 04:57:38,731 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-05 04:57:38,735 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 5708f0f53456b130f41c37899ae0e2e0, NAME => 'TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp 2023-04-05 04:57:38,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-05 04:57:39,145 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 04:57:39,145 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 5708f0f53456b130f41c37899ae0e2e0, disabling compactions & flushes 2023-04-05 04:57:39,145 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:39,145 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:39,145 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. after waiting 0 ms 2023-04-05 04:57:39,146 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:39,146 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:39,146 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 5708f0f53456b130f41c37899ae0e2e0: 2023-04-05 04:57:39,152 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-05 04:57:39,155 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670659155"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680670659155"}]},"ts":"1680670659155"} 2023-04-05 04:57:39,157 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-05 04:57:39,159 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-05 04:57:39,159 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680670659159"}]},"ts":"1680670659159"} 2023-04-05 04:57:39,160 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-05 04:57:39,163 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5708f0f53456b130f41c37899ae0e2e0, ASSIGN}] 2023-04-05 04:57:39,165 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5708f0f53456b130f41c37899ae0e2e0, ASSIGN 2023-04-05 04:57:39,166 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=5708f0f53456b130f41c37899ae0e2e0, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,38091,1680670569206; forceNewPlan=false, retain=false 2023-04-05 04:57:39,318 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=5708f0f53456b130f41c37899ae0e2e0, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:39,318 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670659318"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680670659318"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680670659318"}]},"ts":"1680670659318"} 2023-04-05 04:57:39,321 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure 5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206}] 2023-04-05 04:57:39,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-05 04:57:39,483 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:39,483 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 5708f0f53456b130f41c37899ae0e2e0, NAME => 'TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0.', STARTKEY => '', ENDKEY => ''} 2023-04-05 04:57:39,483 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:57:39,483 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 04:57:39,483 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:57:39,483 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:57:39,485 INFO [StoreOpener-5708f0f53456b130f41c37899ae0e2e0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:57:39,487 DEBUG [StoreOpener-5708f0f53456b130f41c37899ae0e2e0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A 2023-04-05 04:57:39,488 DEBUG [StoreOpener-5708f0f53456b130f41c37899ae0e2e0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A 2023-04-05 04:57:39,488 INFO [StoreOpener-5708f0f53456b130f41c37899ae0e2e0-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 04:57:39,488 INFO [StoreOpener-5708f0f53456b130f41c37899ae0e2e0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5708f0f53456b130f41c37899ae0e2e0 columnFamilyName A 2023-04-05 04:57:39,489 INFO [StoreOpener-5708f0f53456b130f41c37899ae0e2e0-1] regionserver.HStore(310): Store=5708f0f53456b130f41c37899ae0e2e0/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 04:57:39,489 INFO [StoreOpener-5708f0f53456b130f41c37899ae0e2e0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:57:39,492 DEBUG [StoreOpener-5708f0f53456b130f41c37899ae0e2e0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B 2023-04-05 04:57:39,492 DEBUG [StoreOpener-5708f0f53456b130f41c37899ae0e2e0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B 2023-04-05 04:57:39,492 INFO [StoreOpener-5708f0f53456b130f41c37899ae0e2e0-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 04:57:39,492 INFO [StoreOpener-5708f0f53456b130f41c37899ae0e2e0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5708f0f53456b130f41c37899ae0e2e0 columnFamilyName B 2023-04-05 04:57:39,493 INFO [StoreOpener-5708f0f53456b130f41c37899ae0e2e0-1] regionserver.HStore(310): Store=5708f0f53456b130f41c37899ae0e2e0/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 04:57:39,493 INFO [StoreOpener-5708f0f53456b130f41c37899ae0e2e0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:57:39,496 DEBUG [StoreOpener-5708f0f53456b130f41c37899ae0e2e0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C 2023-04-05 04:57:39,496 DEBUG [StoreOpener-5708f0f53456b130f41c37899ae0e2e0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C 2023-04-05 04:57:39,496 INFO [StoreOpener-5708f0f53456b130f41c37899ae0e2e0-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 04:57:39,497 INFO [StoreOpener-5708f0f53456b130f41c37899ae0e2e0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5708f0f53456b130f41c37899ae0e2e0 columnFamilyName C 2023-04-05 04:57:39,498 INFO [StoreOpener-5708f0f53456b130f41c37899ae0e2e0-1] regionserver.HStore(310): Store=5708f0f53456b130f41c37899ae0e2e0/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 04:57:39,498 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:39,499 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:57:39,499 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:57:39,501 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-05 04:57:39,502 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:57:39,503 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-05 04:57:39,503 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 5708f0f53456b130f41c37899ae0e2e0; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69837436, jitterRate=0.04065889120101929}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-05 04:57:39,503 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 5708f0f53456b130f41c37899ae0e2e0: 2023-04-05 04:57:39,504 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0., pid=37, masterSystemTime=1680670659475 2023-04-05 04:57:39,505 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:39,505 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:39,505 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=5708f0f53456b130f41c37899ae0e2e0, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:39,505 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670659505"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680670659505"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680670659505"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680670659505"}]},"ts":"1680670659505"} 2023-04-05 04:57:39,507 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-05 04:57:39,507 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure 5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 in 185 msec 2023-04-05 04:57:39,508 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-05 04:57:39,509 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=5708f0f53456b130f41c37899ae0e2e0, ASSIGN in 344 msec 2023-04-05 04:57:39,509 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-05 04:57:39,509 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680670659509"}]},"ts":"1680670659509"} 2023-04-05 04:57:39,510 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-05 04:57:39,512 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-05 04:57:39,513 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1960 sec 2023-04-05 04:57:40,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-05 04:57:40,430 INFO [Listener at localhost.localdomain/34579] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-05 04:57:40,432 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x7d243e46 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:57:40,439 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@29738198, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:57:40,441 DEBUG [Listener at localhost.localdomain/34579] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:57:40,444 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52158, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:57:40,445 DEBUG [Listener at localhost.localdomain/34579] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-05 04:57:40,447 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:45626, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-05 04:57:40,448 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-05 04:57:40,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-05 04:57:40,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-05 04:57:40,861 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-05 04:57:40,864 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-05 04:57:40,864 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-05 04:57:40,869 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-05 04:57:40,872 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5708f0f53456b130f41c37899ae0e2e0, REOPEN/MOVE}] 2023-04-05 04:57:40,873 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5708f0f53456b130f41c37899ae0e2e0, REOPEN/MOVE 2023-04-05 04:57:40,874 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=5708f0f53456b130f41c37899ae0e2e0, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:40,874 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670660874"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680670660874"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680670660874"}]},"ts":"1680670660874"} 2023-04-05 04:57:40,875 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure 5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206}] 2023-04-05 04:57:41,028 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:57:41,029 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 5708f0f53456b130f41c37899ae0e2e0, disabling compactions & flushes 2023-04-05 04:57:41,029 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:41,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:41,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. after waiting 0 ms 2023-04-05 04:57:41,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:41,038 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-05 04:57:41,040 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:41,040 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 5708f0f53456b130f41c37899ae0e2e0: 2023-04-05 04:57:41,040 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 5708f0f53456b130f41c37899ae0e2e0 to self. 2023-04-05 04:57:41,042 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:57:41,043 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=5708f0f53456b130f41c37899ae0e2e0, regionState=CLOSED 2023-04-05 04:57:41,043 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670661043"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680670661043"}]},"ts":"1680670661043"} 2023-04-05 04:57:41,046 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-05 04:57:41,046 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure 5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 in 169 msec 2023-04-05 04:57:41,047 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=5708f0f53456b130f41c37899ae0e2e0, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,38091,1680670569206; forceNewPlan=false, retain=true 2023-04-05 04:57:41,197 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=5708f0f53456b130f41c37899ae0e2e0, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:41,198 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670661197"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680670661197"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680670661197"}]},"ts":"1680670661197"} 2023-04-05 04:57:41,199 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure 5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206}] 2023-04-05 04:57:41,358 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:41,359 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 5708f0f53456b130f41c37899ae0e2e0, NAME => 'TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0.', STARTKEY => '', ENDKEY => ''} 2023-04-05 04:57:41,359 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:57:41,359 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 04:57:41,360 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:57:41,360 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:57:41,361 INFO [StoreOpener-5708f0f53456b130f41c37899ae0e2e0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:57:41,362 DEBUG [StoreOpener-5708f0f53456b130f41c37899ae0e2e0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A 2023-04-05 04:57:41,362 DEBUG [StoreOpener-5708f0f53456b130f41c37899ae0e2e0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A 2023-04-05 04:57:41,363 INFO [StoreOpener-5708f0f53456b130f41c37899ae0e2e0-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 04:57:41,363 INFO [StoreOpener-5708f0f53456b130f41c37899ae0e2e0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5708f0f53456b130f41c37899ae0e2e0 columnFamilyName A 2023-04-05 04:57:41,365 INFO [StoreOpener-5708f0f53456b130f41c37899ae0e2e0-1] regionserver.HStore(310): Store=5708f0f53456b130f41c37899ae0e2e0/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 04:57:41,365 INFO [StoreOpener-5708f0f53456b130f41c37899ae0e2e0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:57:41,366 DEBUG [StoreOpener-5708f0f53456b130f41c37899ae0e2e0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B 2023-04-05 04:57:41,367 DEBUG [StoreOpener-5708f0f53456b130f41c37899ae0e2e0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B 2023-04-05 04:57:41,367 INFO [StoreOpener-5708f0f53456b130f41c37899ae0e2e0-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 04:57:41,367 INFO [StoreOpener-5708f0f53456b130f41c37899ae0e2e0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5708f0f53456b130f41c37899ae0e2e0 columnFamilyName B 2023-04-05 04:57:41,368 INFO [StoreOpener-5708f0f53456b130f41c37899ae0e2e0-1] regionserver.HStore(310): Store=5708f0f53456b130f41c37899ae0e2e0/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 04:57:41,368 INFO [StoreOpener-5708f0f53456b130f41c37899ae0e2e0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:57:41,369 DEBUG [StoreOpener-5708f0f53456b130f41c37899ae0e2e0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C 2023-04-05 04:57:41,369 DEBUG [StoreOpener-5708f0f53456b130f41c37899ae0e2e0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C 2023-04-05 04:57:41,369 INFO [StoreOpener-5708f0f53456b130f41c37899ae0e2e0-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 04:57:41,370 INFO [StoreOpener-5708f0f53456b130f41c37899ae0e2e0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5708f0f53456b130f41c37899ae0e2e0 columnFamilyName C 2023-04-05 04:57:41,370 INFO [StoreOpener-5708f0f53456b130f41c37899ae0e2e0-1] regionserver.HStore(310): Store=5708f0f53456b130f41c37899ae0e2e0/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 04:57:41,371 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:41,372 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:57:41,373 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:57:41,376 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-05 04:57:41,377 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:57:41,379 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 5708f0f53456b130f41c37899ae0e2e0; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67687378, jitterRate=0.008620530366897583}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-05 04:57:41,379 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 5708f0f53456b130f41c37899ae0e2e0: 2023-04-05 04:57:41,379 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0., pid=42, masterSystemTime=1680670661351 2023-04-05 04:57:41,381 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:41,381 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:41,381 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=5708f0f53456b130f41c37899ae0e2e0, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:41,382 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670661381"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680670661381"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680670661381"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680670661381"}]},"ts":"1680670661381"} 2023-04-05 04:57:41,385 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-05 04:57:41,385 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure 5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 in 185 msec 2023-04-05 04:57:41,387 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-05 04:57:41,387 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=5708f0f53456b130f41c37899ae0e2e0, REOPEN/MOVE in 513 msec 2023-04-05 04:57:41,390 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-05 04:57:41,390 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 520 msec 2023-04-05 04:57:41,393 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 942 msec 2023-04-05 04:57:41,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-05 04:57:41,396 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x74a355de to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:57:41,401 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28a98302, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:57:41,402 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x57b6628e to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:57:41,415 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@39d5c003, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:57:41,415 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x5da5030e to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:57:41,424 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@39e3a826, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:57:41,424 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x74d6bf18 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:57:41,436 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@19e7fff, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:57:41,436 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x060c5cba to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:57:41,445 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@47e30d1c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:57:41,446 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x36a9ece3 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:57:41,456 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69fd9cc7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:57:41,457 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x2ded66ad to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:57:41,465 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54da4907, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:57:41,466 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x14d061d2 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:57:41,475 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52cea5eb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:57:41,476 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x228ff1a4 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:57:41,494 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3546dad6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:57:41,495 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x143a1290 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:57:41,503 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4965c362, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:57:41,513 DEBUG [hconnection-0x2e2dc524-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:57:41,513 DEBUG [hconnection-0x2c9af299-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:57:41,513 DEBUG [hconnection-0x18bee808-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:57:41,514 DEBUG [hconnection-0x67152022-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:57:41,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:41,514 DEBUG [hconnection-0x3e592781-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:57:41,517 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54542, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:57:41,519 DEBUG [hconnection-0x405f7558-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:57:41,517 DEBUG [hconnection-0x4566c055-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:57:41,517 DEBUG [hconnection-0x23ceb1ff-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:57:41,517 DEBUG [hconnection-0x58cec25d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:57:41,517 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54552, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:57:41,519 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54556, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:57:41,519 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54544, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:57:41,519 DEBUG [hconnection-0x454641b7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:57:41,520 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54566, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:57:41,521 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54568, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:57:41,524 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54594, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:57:41,525 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54616, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:57:41,526 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54584, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:57:41,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:41,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:41,530 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:41,530 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:41,531 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54608, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:57:41,531 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:41,531 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:41,531 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:41,531 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:41,539 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:41,539 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:41,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:41,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:41,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:41,540 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:41,540 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:41,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:41,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:41,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:41,541 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:41,541 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:41,541 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:41,541 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:41,541 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:41,541 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:41,542 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:41,542 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:41,542 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:41,542 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:41,543 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5708f0f53456b130f41c37899ae0e2e0 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-05 04:57:41,544 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=A 2023-04-05 04:57:41,544 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:41,544 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=B 2023-04-05 04:57:41,544 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:41,544 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=C 2023-04-05 04:57:41,544 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:41,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:57:41,548 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:41,565 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670661554, max timestamp=1680670661563Num uniques 859; 2023-04-05 04:57:41,565 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:57:41,565 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670661554, max timestamp=1680670661563Num uniques 859; 2023-04-05 04:57:41,565 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:57:41,566 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670661554, max timestamp=1680670661563Num uniques 859; 2023-04-05 04:57:41,566 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:57:41,566 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405bbf41759205f419b8335e0fde99e9818 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405bbf41759205f419b8335e0fde99e9818 2023-04-05 04:57:41,579 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:41,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:41,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54552 deadline: 1680670721577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:41,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54542 deadline: 1680670721577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:41,579 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:41,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54584 deadline: 1680670721578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:41,581 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:41,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54608 deadline: 1680670721579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:41,581 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:41,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670721579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:41,683 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:41,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54552 deadline: 1680670721680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:41,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:41,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54542 deadline: 1680670721680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:41,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:41,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54584 deadline: 1680670721680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:41,686 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:41,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54608 deadline: 1680670721682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:41,686 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:41,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670721682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:41,886 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:41,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54552 deadline: 1680670721885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:41,886 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:41,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54542 deadline: 1680670721885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:41,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:41,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54584 deadline: 1680670721886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:41,890 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:41,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54608 deadline: 1680670721886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:41,890 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:41,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670721887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:41,976 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=13, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/528eb5f66cfd46b4aec10a4453ab1425 2023-04-05 04:57:42,189 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:42,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54542 deadline: 1680670722187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:42,189 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:42,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54552 deadline: 1680670722188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:42,190 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:42,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54584 deadline: 1680670722188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:42,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:42,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670722191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:42,196 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:42,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54608 deadline: 1680670722193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:42,385 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/33faa2f9280b46fe9f74f012a6d4fadc 2023-04-05 04:57:42,694 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:42,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54584 deadline: 1680670722692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:42,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:42,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54552 deadline: 1680670722692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:42,696 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:42,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54542 deadline: 1680670722693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:42,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:42,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670722697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:42,700 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:42,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54608 deadline: 1680670722698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:42,793 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/0cac5d39573c4344b6718fcb29b77c34 2023-04-05 04:57:42,797 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/528eb5f66cfd46b4aec10a4453ab1425 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/528eb5f66cfd46b4aec10a4453ab1425 2023-04-05 04:57:42,800 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/528eb5f66cfd46b4aec10a4453ab1425, entries=100, sequenceid=13, filesize=18.4 K 2023-04-05 04:57:42,801 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/33faa2f9280b46fe9f74f012a6d4fadc as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/33faa2f9280b46fe9f74f012a6d4fadc 2023-04-05 04:57:42,804 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/33faa2f9280b46fe9f74f012a6d4fadc, entries=100, sequenceid=13, filesize=9.3 K 2023-04-05 04:57:42,804 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/0cac5d39573c4344b6718fcb29b77c34 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/0cac5d39573c4344b6718fcb29b77c34 2023-04-05 04:57:42,810 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/0cac5d39573c4344b6718fcb29b77c34, entries=100, sequenceid=13, filesize=9.3 K 2023-04-05 04:57:42,811 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 5708f0f53456b130f41c37899ae0e2e0 in 1268ms, sequenceid=13, compaction requested=false 2023-04-05 04:57:42,811 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5708f0f53456b130f41c37899ae0e2e0: 2023-04-05 04:57:42,811 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:42,812 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:42,812 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:42,812 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:42,812 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:42,812 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:42,813 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:42,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:42,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:42,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:42,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:42,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:42,814 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:42,814 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:42,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:42,814 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:42,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:42,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:42,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:42,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4905daf4[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:42,815 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:42,815 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:42,815 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:42,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:42,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:42,815 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:42,815 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:42,815 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:42,815 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:42,815 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:42,816 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:42,816 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:42,816 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:42,816 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:42,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:42,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:42,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:42,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:42,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:42,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:42,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:42,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:42,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:42,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:42,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:42,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:42,818 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:42,818 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:42,818 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:42,818 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:42,818 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:42,818 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:42,818 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:42,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:42,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:42,818 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:42,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:42,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:42,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:42,819 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:42,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:42,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:42,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:42,821 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:42,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:42,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:42,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:42,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:42,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:42,823 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:42,823 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:42,823 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:42,823 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:42,823 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:42,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:42,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:42,823 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:42,823 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:42,823 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:42,823 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:42,823 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:42,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:42,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:42,825 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:42,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:42,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:42,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:42,923 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:42,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:42,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:42,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:42,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:42,926 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:42,926 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:42,926 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:42,926 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:42,926 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:42,926 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:42,927 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:42,927 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:42,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:42,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:42,927 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:42,927 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:42,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:42,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:42,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:42,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:42,928 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:42,928 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:42,928 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:42,928 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:42,928 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:42,928 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:42,928 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:42,928 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:42,928 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:42,928 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:42,928 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5708f0f53456b130f41c37899ae0e2e0 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-05 04:57:42,928 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=A 2023-04-05 04:57:42,929 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:42,929 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=B 2023-04-05 04:57:42,929 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:42,929 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=C 2023-04-05 04:57:42,929 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:43,123 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-05 04:57:43,341 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304054b28fa3726014c2ca19d8ad49649e68d to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304054b28fa3726014c2ca19d8ad49649e68d 2023-04-05 04:57:43,345 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/1c8bf2ff48734f41b9afcec9643dcbd2 2023-04-05 04:57:43,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:57:43,699 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:43,707 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:43,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54608 deadline: 1680670723704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:43,709 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:43,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54584 deadline: 1680670723705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:43,712 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:43,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54552 deadline: 1680670723707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:43,712 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:43,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54542 deadline: 1680670723708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:43,712 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:43,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670723708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:43,755 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/a97c9e1c033a49f78909610b29b17e43 2023-04-05 04:57:43,812 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:43,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54608 deadline: 1680670723808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:43,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:43,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54584 deadline: 1680670723810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:43,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:43,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54552 deadline: 1680670723813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:43,814 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:43,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54542 deadline: 1680670723813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:44,022 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:44,022 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:44,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54608 deadline: 1680670724014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:44,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54542 deadline: 1680670724015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:44,022 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:44,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54584 deadline: 1680670724015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:44,022 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:44,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54552 deadline: 1680670724015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:44,163 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/6a559d5bfa4f4a949f4463fe00bb09b3 2023-04-05 04:57:44,167 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/1c8bf2ff48734f41b9afcec9643dcbd2 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/1c8bf2ff48734f41b9afcec9643dcbd2 2023-04-05 04:57:44,170 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/1c8bf2ff48734f41b9afcec9643dcbd2, entries=150, sequenceid=50, filesize=25.1 K 2023-04-05 04:57:44,171 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/a97c9e1c033a49f78909610b29b17e43 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/a97c9e1c033a49f78909610b29b17e43 2023-04-05 04:57:44,174 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/a97c9e1c033a49f78909610b29b17e43, entries=150, sequenceid=50, filesize=11.6 K 2023-04-05 04:57:44,175 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/6a559d5bfa4f4a949f4463fe00bb09b3 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/6a559d5bfa4f4a949f4463fe00bb09b3 2023-04-05 04:57:44,180 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/6a559d5bfa4f4a949f4463fe00bb09b3, entries=150, sequenceid=50, filesize=11.6 K 2023-04-05 04:57:44,180 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=46.96 KB/48090 for 5708f0f53456b130f41c37899ae0e2e0 in 1252ms, sequenceid=50, compaction requested=false 2023-04-05 04:57:44,180 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5708f0f53456b130f41c37899ae0e2e0: 2023-04-05 04:57:44,181 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:44,181 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:44,181 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:44,181 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:44,181 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:44,181 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:44,181 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:44,181 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:44,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:44,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:44,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:44,182 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:44,182 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:44,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:44,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:44,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:44,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:44,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:44,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:44,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48d000a8[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:44,183 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:44,183 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:44,184 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:44,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:44,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:44,184 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:44,184 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:44,184 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:44,184 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:44,184 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:44,184 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:44,184 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:44,184 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:44,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:44,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:44,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:44,184 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:44,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:44,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:44,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:44,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:44,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:44,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:44,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:44,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:44,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:44,186 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:44,186 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:44,186 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:44,186 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:44,186 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:44,187 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:44,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:44,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:44,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:44,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:44,188 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:44,188 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:44,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:44,188 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:44,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:44,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:44,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:44,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:44,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:44,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:44,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:44,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:44,189 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:44,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:44,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:44,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:44,190 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:44,190 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:44,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:44,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:44,191 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:44,191 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:44,191 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:44,191 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:44,191 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:44,191 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:44,191 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:44,191 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:44,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:44,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:44,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:44,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:44,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:44,191 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:44,191 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:44,291 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:44,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:44,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:44,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:44,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:44,294 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:44,294 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:44,294 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:44,294 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:44,295 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:44,295 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:44,295 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:44,295 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:44,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:44,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:44,295 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:44,295 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:44,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:44,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:44,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:44,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:44,296 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:44,296 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:44,296 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:44,296 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:44,296 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:44,297 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:44,297 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:44,297 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:44,297 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:44,297 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:44,297 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5708f0f53456b130f41c37899ae0e2e0 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 04:57:44,297 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=A 2023-04-05 04:57:44,297 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:44,297 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=B 2023-04-05 04:57:44,297 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:44,297 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=C 2023-04-05 04:57:44,297 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:44,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:57:44,325 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:44,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680670664339, max timestamp=1680670664350Num uniques 859; 2023-04-05 04:57:44,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 04:57:44,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680670664339, max timestamp=1680670664350Num uniques 859; 2023-04-05 04:57:44,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 04:57:44,353 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680670664339, max timestamp=1680670664350Num uniques 859; 2023-04-05 04:57:44,353 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 04:57:44,395 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:44,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54608 deadline: 1680670724394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:44,400 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:44,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54542 deadline: 1680670724395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:44,400 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:44,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54584 deadline: 1680670724395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:44,401 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:44,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54552 deadline: 1680670724395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:44,503 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:44,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54608 deadline: 1680670724496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:44,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:44,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54542 deadline: 1680670724501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:44,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:44,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54584 deadline: 1680670724501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:44,507 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:44,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54552 deadline: 1680670724501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:44,709 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:44,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54608 deadline: 1680670724705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:44,710 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:44,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54542 deadline: 1680670724707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:44,710 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:44,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54552 deadline: 1680670724708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:44,713 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:44,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54584 deadline: 1680670724708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:44,716 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405aef59b91d37b46f0bdb22abca4f931b7 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405aef59b91d37b46f0bdb22abca4f931b7 2023-04-05 04:57:44,722 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=60, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/6280e137855a46e0a7e3d6e8b8cca24a 2023-04-05 04:57:45,017 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:45,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54608 deadline: 1680670725012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:45,017 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:45,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54542 deadline: 1680670725013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:45,018 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:45,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54552 deadline: 1680670725013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:45,018 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:45,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54584 deadline: 1680670725013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:45,130 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/014a77a107964d94a4a26a9846973ac0 2023-04-05 04:57:45,521 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:45,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54552 deadline: 1680670725518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:45,522 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:45,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54584 deadline: 1680670725518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:45,522 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:45,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54608 deadline: 1680670725520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:45,527 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:45,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54542 deadline: 1680670725522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:45,539 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/533eca10fbbe42e39d786d2fd1ff00c1 2023-04-05 04:57:45,542 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/6280e137855a46e0a7e3d6e8b8cca24a as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/6280e137855a46e0a7e3d6e8b8cca24a 2023-04-05 04:57:45,545 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/6280e137855a46e0a7e3d6e8b8cca24a, entries=150, sequenceid=60, filesize=25.1 K 2023-04-05 04:57:45,545 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/014a77a107964d94a4a26a9846973ac0 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/014a77a107964d94a4a26a9846973ac0 2023-04-05 04:57:45,549 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/014a77a107964d94a4a26a9846973ac0, entries=150, sequenceid=60, filesize=11.6 K 2023-04-05 04:57:45,549 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/533eca10fbbe42e39d786d2fd1ff00c1 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/533eca10fbbe42e39d786d2fd1ff00c1 2023-04-05 04:57:45,552 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/533eca10fbbe42e39d786d2fd1ff00c1, entries=150, sequenceid=60, filesize=11.6 K 2023-04-05 04:57:45,552 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=147.60 KB/151140 for 5708f0f53456b130f41c37899ae0e2e0 in 1255ms, sequenceid=60, compaction requested=true 2023-04-05 04:57:45,552 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5708f0f53456b130f41c37899ae0e2e0: 2023-04-05 04:57:45,552 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:45,553 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:45,553 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:45,553 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:45,553 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:45,553 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:45,554 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:45,554 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:45,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:45,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:45,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:45,554 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:45,554 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:45,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:45,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:45,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:45,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:45,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:45,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:45,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52534cd[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:45,555 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:45,555 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:45,556 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:45,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:45,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:45,556 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:45,556 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:45,556 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:45,556 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:45,556 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:45,556 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:45,556 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:45,556 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:45,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:45,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:45,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:45,556 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:45,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:45,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:45,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:45,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:45,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:45,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:45,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:45,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:45,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:45,558 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:45,558 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:45,558 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:45,558 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:45,558 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:45,559 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:45,559 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:45,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:45,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:45,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:45,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:45,560 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:45,560 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:45,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:45,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:45,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:45,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:45,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:45,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:45,560 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:45,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:45,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:45,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:45,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:45,562 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:45,562 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:45,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:45,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:45,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:45,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:45,562 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:45,562 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:45,562 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:45,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:45,563 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:45,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:45,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:45,563 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:45,563 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:45,563 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:45,563 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:45,563 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:45,563 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:45,663 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:45,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:45,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:45,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:45,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:45,666 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:45,666 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:45,666 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:45,666 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:45,666 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:45,666 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:45,667 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:45,667 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:45,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:45,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:45,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:45,667 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:45,667 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:45,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:45,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:45,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:45,668 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:45,668 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:45,668 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:45,668 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:45,668 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:45,668 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:45,668 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:45,668 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:45,668 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:45,668 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:45,668 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5708f0f53456b130f41c37899ae0e2e0 3/3 column families, dataSize=147.60 KB heapSize=415.45 KB 2023-04-05 04:57:45,669 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=A 2023-04-05 04:57:45,669 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:45,669 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=B 2023-04-05 04:57:45,669 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:45,669 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=C 2023-04-05 04:57:45,669 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:45,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:57:45,735 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:45,762 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:45,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670725757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:45,868 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:45,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670725863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,072 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:46,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670726070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,088 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405b000c6c7623642a8945b3a72203627b6 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405b000c6c7623642a8945b3a72203627b6 2023-04-05 04:57:46,100 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=97, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/0e0d17cef28d4b34b14c65c2dc7dad68 2023-04-05 04:57:46,108 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/6de5715de2d140098a4afb6c4084062b 2023-04-05 04:57:46,378 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:46,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670726373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,518 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/adbf54ec6fdb45acbd92fdb0ad316369 2023-04-05 04:57:46,521 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/0e0d17cef28d4b34b14c65c2dc7dad68 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/0e0d17cef28d4b34b14c65c2dc7dad68 2023-04-05 04:57:46,524 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/0e0d17cef28d4b34b14c65c2dc7dad68, entries=150, sequenceid=97, filesize=25.1 K 2023-04-05 04:57:46,525 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/6de5715de2d140098a4afb6c4084062b as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/6de5715de2d140098a4afb6c4084062b 2023-04-05 04:57:46,528 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/6de5715de2d140098a4afb6c4084062b, entries=150, sequenceid=97, filesize=11.6 K 2023-04-05 04:57:46,528 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/adbf54ec6fdb45acbd92fdb0ad316369 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/adbf54ec6fdb45acbd92fdb0ad316369 2023-04-05 04:57:46,535 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:46,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54608 deadline: 1680670726526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,544 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/adbf54ec6fdb45acbd92fdb0ad316369, entries=150, sequenceid=97, filesize=11.6 K 2023-04-05 04:57:46,544 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~415.41 KB/425376, currentSize=33.54 KB/34350 for 5708f0f53456b130f41c37899ae0e2e0 in 876ms, sequenceid=97, compaction requested=true 2023-04-05 04:57:46,545 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5708f0f53456b130f41c37899ae0e2e0: 2023-04-05 04:57:46,545 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:46,545 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:46,545 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:46,545 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,545 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:46,545 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:46,546 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:46,546 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:46,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:46,546 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:46,546 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:46,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:46,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:46,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:46,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:46,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:46,551 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:46,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7af94b6[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:46,551 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:46,555 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:46,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:46,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:46,555 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:46,555 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:46,555 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,555 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:46,555 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:46,561 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:46,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:46,562 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:46,562 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:46,562 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:46,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:46,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:46,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:46,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:46,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:46,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:46,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:46,564 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:46,564 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:46,564 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:46,564 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:46,564 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:46,568 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:46,568 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:46,568 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:46,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:46,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:46,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:46,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:46,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:46,569 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:46,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:46,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:46,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:46,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:46,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:46,570 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:46,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:46,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:46,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,572 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,572 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:46,572 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:46,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:46,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:46,572 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:46,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:46,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:46,572 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:46,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:46,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:46,572 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:46,572 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:46,572 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:46,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:46,572 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,572 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:46,572 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:46,572 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:46,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:57:46,576 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5708f0f53456b130f41c37899ae0e2e0 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 04:57:46,576 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=A 2023-04-05 04:57:46,576 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:46,576 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=B 2023-04-05 04:57:46,576 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:46,576 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=C 2023-04-05 04:57:46,576 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:46,630 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670666584, max timestamp=1680670666629Num uniques 859; 2023-04-05 04:57:46,630 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:57:46,630 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670666584, max timestamp=1680670666629Num uniques 859; 2023-04-05 04:57:46,630 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670666584, max timestamp=1680670666629Num uniques 859; 2023-04-05 04:57:46,630 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:57:46,631 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:57:46,672 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:46,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:46,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:46,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:46,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:46,676 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:46,676 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:46,676 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:46,676 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:46,676 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:46,676 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:46,677 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:46,677 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:46,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:46,677 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,677 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:46,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:46,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:46,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:46,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:46,677 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:46,678 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:46,678 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:46,678 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:46,678 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:46,678 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:46,682 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:46,682 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:46,682 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:46,682 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:46,682 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:46,682 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:46,682 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:46,682 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:46,682 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,682 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:46,682 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:46,683 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,683 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:46,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:46,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:46,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:46,683 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:46,684 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:46,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:46,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:46,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:46,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:46,685 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:46,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@601a2ccd[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:46,685 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:46,685 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:46,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:46,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:46,685 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:46,685 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:46,685 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:46,686 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,686 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:46,686 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:46,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:46,686 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:46,686 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:46,686 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:46,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:46,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:46,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:46,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:46,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:46,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:46,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:46,688 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:46,688 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:46,688 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:46,688 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:46,688 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:46,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:46,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54542 deadline: 1680670726674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,689 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:46,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54584 deadline: 1680670726675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,689 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:46,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54552 deadline: 1680670726676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,690 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:46,690 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:46,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:46,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:46,690 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:46,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:46,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:46,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:46,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:46,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:46,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:46,691 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,691 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:46,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:46,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:46,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:46,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,693 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:46,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:46,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:46,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:46,694 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:46,694 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:46,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:46,698 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:46,698 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:46,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:46,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:46,698 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,698 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:46,698 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:46,698 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,698 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:46,698 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:46,698 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:46,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:46,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:46,793 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:46,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54542 deadline: 1680670726790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,794 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:46,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:46,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:46,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:46,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:46,798 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:46,798 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:46,798 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:46,798 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:46,798 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:46,798 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:46,799 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,799 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:46,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:46,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:46,799 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,799 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:46,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:46,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:46,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:46,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:46,800 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:46,801 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:46,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54552 deadline: 1680670726790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,802 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:46,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54584 deadline: 1680670726790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,806 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:46,806 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:46,806 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:46,806 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:46,806 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:46,810 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:46,810 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:46,810 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:46,810 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:46,810 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:46,810 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:46,810 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:46,810 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:46,810 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,810 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:46,811 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:46,813 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,813 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:46,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:46,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:46,814 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:46,814 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:46,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:46,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:46,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:46,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:46,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:46,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2807a526[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:46,815 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:46,815 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:46,817 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:46,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:46,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:46,817 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:46,817 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,817 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:46,817 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:46,818 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:46,818 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:46,818 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:46,818 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:46,818 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:46,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:46,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:46,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:46,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:46,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:46,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:46,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:46,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:46,822 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:46,822 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:46,822 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:46,822 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:46,822 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:46,830 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:46,830 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:46,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:46,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:46,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:46,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:46,830 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:46,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:46,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:46,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:46,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:46,831 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:46,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:46,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:46,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:46,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,833 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:46,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:46,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,838 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:46,838 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,839 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:46,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:46,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:46,839 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:46,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:46,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:46,839 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:46,839 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:46,839 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:46,839 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:46,839 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,839 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:46,839 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:46,839 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:46,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:46,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:46,894 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:46,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670726883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,939 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:46,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:46,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:46,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:46,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:46,943 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:46,943 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:46,943 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:46,943 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:46,943 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:46,943 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:46,944 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:46,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:46,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:46,944 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:46,945 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,945 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:46,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:46,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:46,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:46,946 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:46,946 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:46,946 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:46,946 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:46,946 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:46,946 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:46,947 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:46,947 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:46,947 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:46,948 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:46,948 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:46,948 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:46,948 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:46,948 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:46,948 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,948 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:46,948 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:46,953 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:46,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:46,954 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:46,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:46,958 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:46,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:46,958 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:46,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:46,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:46,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:46,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a87fb7[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:46,961 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:46,961 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:46,964 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:46,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:46,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:46,964 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:46,964 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:46,964 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:46,964 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,964 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:46,965 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:46,965 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:46,965 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:46,965 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:46,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:46,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:46,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:46,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:46,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:46,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:46,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:46,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:46,968 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:46,968 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:46,968 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:46,968 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:46,968 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:46,969 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:46,969 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:46,969 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:46,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:46,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:46,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:46,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:46,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:46,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:46,969 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:46,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:46,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:46,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:46,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:46,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:46,972 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:46,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:46,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,974 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:46,974 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,974 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:46,974 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:46,974 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:46,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:46,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:46,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:46,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:46,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:46,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:46,974 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:46,974 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:46,974 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:46,974 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:46,974 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:46,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:46,974 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:46,975 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:46,990 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304053d6be64e99764849a504083e3cc72e7a to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304053d6be64e99764849a504083e3cc72e7a 2023-04-05 04:57:47,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:47,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54542 deadline: 1680670726995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,011 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:47,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54552 deadline: 1680670727004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,011 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:47,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54584 deadline: 1680670727004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,075 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:47,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:47,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:47,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:47,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:47,091 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:47,091 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:47,091 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:47,091 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:47,091 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,091 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:47,092 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,092 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:47,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:47,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:47,093 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,093 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:47,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:47,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:47,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:47,096 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:47,098 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:47,098 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:47,098 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:47,098 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:47,098 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:47,106 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:47,106 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:47,113 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:47,114 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:47,114 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:47,114 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:47,114 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:47,114 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:47,114 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,114 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:47,114 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:47,115 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,115 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:47,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:47,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:47,116 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,116 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:47,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:47,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:47,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:47,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f5951b0[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:47,120 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:47,120 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,126 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,127 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:47,127 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:47,127 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:47,127 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,127 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:47,130 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:47,130 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:47,130 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:47,131 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:47,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:47,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:47,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:47,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:47,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:47,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:47,137 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:47,137 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:47,137 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:47,137 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:47,137 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:47,143 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,143 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:47,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:47,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:47,144 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,144 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:47,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:47,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:47,145 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:47,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,147 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:47,147 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:47,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:47,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:47,147 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,147 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:47,147 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:47,147 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,147 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:47,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:47,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:47,147 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,147 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,147 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:47,148 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:47,148 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:47,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:47,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:47,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:47,248 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:47,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:47,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:47,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:47,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:47,252 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:47,252 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:47,252 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:47,252 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:47,253 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,253 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:47,253 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:47,253 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:47,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:47,254 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,254 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:47,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:47,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:47,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:47,254 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:47,255 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:47,255 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:47,255 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:47,255 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:47,255 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:47,256 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:47,256 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:47,256 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:47,256 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:47,256 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:47,256 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:47,256 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:47,256 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:47,256 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,256 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:47,256 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:47,257 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,257 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:47,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:47,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:47,257 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,258 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:47,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:47,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:47,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:47,259 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:47,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e1476d2[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:47,259 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,259 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,260 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:47,260 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:47,260 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:47,261 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,261 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:47,261 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,261 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:47,261 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:47,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:47,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:47,261 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:47,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:47,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:47,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:47,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:47,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:47,264 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:47,264 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:47,264 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:47,264 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:47,264 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:47,265 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,265 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:47,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,265 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:47,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:47,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:47,266 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:47,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:47,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:47,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,267 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:47,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,269 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,269 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:47,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:47,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:47,269 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:47,269 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:47,269 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:47,269 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:47,269 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,269 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,269 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,269 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,269 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:47,269 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:47,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:47,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:47,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:47,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:47,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:47,308 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:47,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54542 deadline: 1680670727302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,321 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:47,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54584 deadline: 1680670727312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,322 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:47,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54552 deadline: 1680670727314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,369 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:47,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:47,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:47,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:47,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:47,377 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:47,377 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:47,378 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:47,378 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:47,378 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,378 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:47,379 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,379 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:47,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:47,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:47,380 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:47,381 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:47,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:47,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:47,381 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:47,386 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:47,386 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:47,386 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:47,386 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:47,386 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:47,387 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:47,387 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:47,387 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:47,387 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:47,387 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:47,387 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:47,387 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:47,388 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:47,388 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,388 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:47,388 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:47,393 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,393 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:47,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:47,394 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,394 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:47,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:47,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:47,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:47,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:47,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b2fd4a3[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:47,396 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:47,396 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,397 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,397 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:47,398 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:47,398 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:47,398 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,398 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:47,398 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,398 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:47,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:47,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:47,398 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:47,398 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:47,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:47,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:47,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:47,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,400 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=107, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/ad3380382bb74b76af2bce799fce724b 2023-04-05 04:57:47,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:47,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:47,400 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:47,400 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:47,400 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:47,400 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:47,400 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:47,411 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,411 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:47,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:47,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,411 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:47,412 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:47,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:47,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:47,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:47,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:47,413 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,416 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,416 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:47,416 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:47,416 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:47,416 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:47,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:47,416 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,416 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,416 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,416 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:47,416 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:47,416 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:47,416 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:47,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:47,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:47,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:47,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:47,516 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:47,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:47,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:47,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:47,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:47,522 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:47,522 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:47,522 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:47,523 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:47,523 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,523 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:47,524 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:47,524 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:47,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:47,524 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,524 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:47,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:47,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:47,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:47,525 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:47,525 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:47,525 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:47,525 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:47,525 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:47,525 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:47,525 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:47,525 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:47,525 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:47,525 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:47,526 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:47,526 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:47,526 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:47,526 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:47,526 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,526 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:47,526 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:47,527 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,528 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,528 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,528 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:47,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:47,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:47,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:47,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:47,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:47,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:47,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c3d5db6[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:47,529 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:47,529 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,530 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,530 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:47,530 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:47,530 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:47,530 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,530 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:47,531 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:47,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:47,531 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:47,531 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:47,531 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:47,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:47,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:47,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:47,535 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:47,535 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:47,535 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:47,535 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:47,535 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:47,536 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,536 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:47,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:47,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,536 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:47,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:47,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:47,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:47,537 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:47,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,538 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:47,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,541 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:47,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:47,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:47,541 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,541 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:47,541 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:47,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:47,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:47,541 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:47,541 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:47,541 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,541 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:47,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:47,541 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,541 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:47,541 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,541 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:47,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:47,641 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:47,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:47,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:47,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:47,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:47,644 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:47,644 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:47,644 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:47,644 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:47,645 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,645 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:47,645 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:47,645 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:47,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:47,646 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,646 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:47,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:47,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:47,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:47,646 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:47,647 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:47,647 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:47,647 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:47,647 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:47,647 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:47,647 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:47,647 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:47,647 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:47,647 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:47,647 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:47,647 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:47,647 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:47,647 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:47,647 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,647 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:47,647 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:47,648 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,648 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:47,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:47,649 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,649 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:47,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:47,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:47,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29dc4b50[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:47,651 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:47,651 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,652 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,652 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:47,652 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:47,652 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,652 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:47,652 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:47,653 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:47,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:47,653 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:47,653 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:47,653 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:47,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:47,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:47,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:47,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:47,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:47,655 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:47,655 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:47,655 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:47,655 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:47,655 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:47,656 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,656 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:47,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,656 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:47,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:47,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:47,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:47,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:47,657 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:47,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,658 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:47,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,659 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,659 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:47,659 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:47,659 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,659 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,659 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,659 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,659 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:47,659 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:47,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:47,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:47,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:47,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:47,660 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:47,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:47,660 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:47,660 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:47,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:47,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:47,760 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:47,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:47,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:47,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:47,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:47,764 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:47,764 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:47,764 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:47,764 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:47,765 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,765 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:47,766 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,766 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:47,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:47,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:47,766 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,766 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:47,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:47,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:47,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:47,767 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:47,767 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:47,767 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:47,767 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:47,767 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:47,767 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:47,767 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:47,767 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:47,767 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:47,767 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:47,767 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:47,767 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:47,767 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:47,767 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:47,767 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,767 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:47,767 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:47,768 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,768 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:47,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:47,769 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,769 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:47,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:47,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:47,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:47,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:47,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44131a5f[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:47,769 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:47,770 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,770 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,770 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:47,770 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:47,770 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,770 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:47,770 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:47,772 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,772 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:47,772 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:47,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:47,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:47,772 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:47,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:47,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:47,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:47,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:47,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:47,775 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:47,775 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:47,775 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:47,775 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:47,775 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:47,776 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,776 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:47,776 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,776 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:47,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:47,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:47,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:47,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:47,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:47,778 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:47,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,779 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,779 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:47,779 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:47,779 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:47,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:47,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:47,779 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:47,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:47,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:47,779 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:47,779 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,780 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,780 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,780 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,780 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:47,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:47,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:47,780 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:47,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:47,810 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/8821044e6c7648c2a3c87ea8ac08ab66 2023-04-05 04:57:47,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:47,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54542 deadline: 1680670727809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,827 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:47,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54552 deadline: 1680670727823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,827 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:47,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54584 deadline: 1680670727825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,880 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:47,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:47,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:47,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:47,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:47,883 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:47,883 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:47,883 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:47,883 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:47,884 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,885 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:47,892 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:47,892 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:47,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:47,893 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,893 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:47,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:47,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:47,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:47,893 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:47,894 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:47,894 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:47,894 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:47,894 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:47,894 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:47,894 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:47,894 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:47,894 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:47,894 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:47,894 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:47,894 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:47,894 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:47,894 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:47,894 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,894 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:47,894 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:47,895 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,895 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:47,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:47,895 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,895 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:47,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:47,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:47,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:47,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:47,897 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:47,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@613c44b2[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:47,897 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,897 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,897 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,897 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:47,897 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:47,897 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:47,898 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:47,899 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:47,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:47,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:47,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:47,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:47,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,901 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:47,901 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:47,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,901 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:47,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:47,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:47,901 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:47,901 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:47,901 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:47,901 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:47,901 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:47,902 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,902 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,902 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:47,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:47,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:47,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,902 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:47,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:47,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:47,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:47,903 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,905 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:47,905 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,905 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:47,905 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:47,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:47,905 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:47,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:47,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:47,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:47,905 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:47,905 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:47,905 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:47,905 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:47,905 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:47,905 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:47,905 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:47,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:47,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:47,907 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:47,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670727896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,005 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:48,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:48,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:48,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:48,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:48,010 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:48,010 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:48,010 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:48,010 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:48,010 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:48,010 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:48,011 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:48,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:48,011 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:48,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:48,012 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,012 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:48,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:48,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:48,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:48,012 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:48,013 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:48,013 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:48,013 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:48,013 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:48,013 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:48,014 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:48,014 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:48,014 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:48,014 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:48,014 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:48,014 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:48,014 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:48,014 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:48,014 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,014 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:48,014 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:48,015 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:48,015 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:48,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:48,015 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:48,015 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:48,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:48,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:48,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:48,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:48,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:48,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@179a75ce[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:48,017 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:48,017 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:48,017 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:48,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:48,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:48,017 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:48,017 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:48,017 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:48,018 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,018 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:48,018 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:48,018 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:48,018 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,018 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:48,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:48,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:48,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:48,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:48,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:48,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:48,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:48,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:48,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:48,021 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:48,021 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:48,021 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:48,021 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:48,021 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:48,023 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:48,023 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:48,023 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:48,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:48,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:48,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:48,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:48,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:48,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:48,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:48,023 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:48,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:48,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:48,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:48,025 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:48,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:48,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:48,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,026 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,026 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:48,026 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:48,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:48,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:48,026 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:48,027 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:48,027 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:48,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:48,027 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:48,027 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:48,027 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:48,027 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:48,028 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:48,028 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:48,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:48,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:48,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:48,127 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:48,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:48,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:48,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:48,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:48,130 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:48,130 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:48,130 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:48,130 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:48,131 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:48,131 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:48,131 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:48,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:48,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:48,131 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:48,132 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,132 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:48,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:48,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:48,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:48,132 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:48,132 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:48,132 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:48,133 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:48,133 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:48,133 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:48,133 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:48,133 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:48,133 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:48,133 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:48,133 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:48,133 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:48,133 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:48,133 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:48,133 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,133 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:48,133 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:48,134 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:48,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:48,134 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:48,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:48,135 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:48,135 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:48,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:48,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:48,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:48,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:48,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c573aa6[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:48,136 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:48,136 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:48,136 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:48,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:48,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:48,136 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:48,136 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:48,136 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,137 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:48,136 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:48,137 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,137 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:48,137 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:48,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:48,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:48,137 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:48,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:48,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:48,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:48,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:48,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:48,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:48,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:48,140 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:48,140 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:48,140 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:48,140 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:48,140 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:48,141 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:48,141 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:48,141 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:48,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:48,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:48,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:48,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:48,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:48,141 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:48,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:48,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:48,142 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:48,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:48,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:48,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:48,145 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,145 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:48,145 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:48,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:48,145 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:48,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:48,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:48,145 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:48,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:48,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:48,145 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:48,145 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,145 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:48,145 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:48,145 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:48,145 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:48,145 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:48,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:48,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:48,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:48,230 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/75e7b6ff42ab4113b4e2e6205999bb40 2023-04-05 04:57:48,234 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/ad3380382bb74b76af2bce799fce724b as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/ad3380382bb74b76af2bce799fce724b 2023-04-05 04:57:48,237 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/ad3380382bb74b76af2bce799fce724b, entries=200, sequenceid=107, filesize=31.9 K 2023-04-05 04:57:48,238 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/8821044e6c7648c2a3c87ea8ac08ab66 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/8821044e6c7648c2a3c87ea8ac08ab66 2023-04-05 04:57:48,243 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/8821044e6c7648c2a3c87ea8ac08ab66, entries=150, sequenceid=107, filesize=11.6 K 2023-04-05 04:57:48,243 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/75e7b6ff42ab4113b4e2e6205999bb40 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/75e7b6ff42ab4113b4e2e6205999bb40 2023-04-05 04:57:48,245 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:48,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:48,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:48,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:48,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:48,248 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:48,248 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:48,248 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:48,248 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:48,251 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:48,251 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:48,254 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/75e7b6ff42ab4113b4e2e6205999bb40, entries=150, sequenceid=107, filesize=11.6 K 2023-04-05 04:57:48,255 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:48,255 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:48,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:48,255 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 5708f0f53456b130f41c37899ae0e2e0 in 1679ms, sequenceid=107, compaction requested=true 2023-04-05 04:57:48,255 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5708f0f53456b130f41c37899ae0e2e0: 2023-04-05 04:57:48,256 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:57:48,256 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 04:57:48,256 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 04:57:48,256 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:57:48,256 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:57:48,258 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:48,258 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:48,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:48,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:48,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:48,259 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:48,262 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:48,262 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:48,262 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:48,262 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:48,262 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:48,264 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57086 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 04:57:48,264 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): 5708f0f53456b130f41c37899ae0e2e0/B is initiating minor compaction (all files) 2023-04-05 04:57:48,264 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5708f0f53456b130f41c37899ae0e2e0/B in TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:48,264 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/33faa2f9280b46fe9f74f012a6d4fadc, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/a97c9e1c033a49f78909610b29b17e43, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/014a77a107964d94a4a26a9846973ac0, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/6de5715de2d140098a4afb6c4084062b, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/8821044e6c7648c2a3c87ea8ac08ab66] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp, totalSize=55.7 K 2023-04-05 04:57:48,264 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 33faa2f9280b46fe9f74f012a6d4fadc, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680670661538 2023-04-05 04:57:48,265 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting a97c9e1c033a49f78909610b29b17e43, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680670661575 2023-04-05 04:57:48,265 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 014a77a107964d94a4a26a9846973ac0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680670663701 2023-04-05 04:57:48,265 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 6de5715de2d140098a4afb6c4084062b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1680670664388 2023-04-05 04:57:48,266 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 8821044e6c7648c2a3c87ea8ac08ab66, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680670665756 2023-04-05 04:57:48,271 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 128722 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 04:57:48,271 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): 5708f0f53456b130f41c37899ae0e2e0/A is initiating minor compaction (all files) 2023-04-05 04:57:48,271 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5708f0f53456b130f41c37899ae0e2e0/A in TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:48,271 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/528eb5f66cfd46b4aec10a4453ab1425, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/1c8bf2ff48734f41b9afcec9643dcbd2, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/6280e137855a46e0a7e3d6e8b8cca24a, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/0e0d17cef28d4b34b14c65c2dc7dad68, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/ad3380382bb74b76af2bce799fce724b] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp, totalSize=125.7 K 2023-04-05 04:57:48,272 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 528eb5f66cfd46b4aec10a4453ab1425, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680670661538 2023-04-05 04:57:48,272 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 1c8bf2ff48734f41b9afcec9643dcbd2, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680670661575 2023-04-05 04:57:48,273 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 6280e137855a46e0a7e3d6e8b8cca24a, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680670663701 2023-04-05 04:57:48,273 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 0e0d17cef28d4b34b14c65c2dc7dad68, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1680670664388 2023-04-05 04:57:48,274 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting ad3380382bb74b76af2bce799fce724b, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680670665742 2023-04-05 04:57:48,274 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:48,274 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:48,274 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:48,274 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:48,274 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5708f0f53456b130f41c37899ae0e2e0 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-05 04:57:48,274 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=A 2023-04-05 04:57:48,274 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:48,274 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=B 2023-04-05 04:57:48,275 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:48,275 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=C 2023-04-05 04:57:48,275 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:48,285 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): 5708f0f53456b130f41c37899ae0e2e0#B#compaction#302 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:57:48,308 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5708f0f53456b130f41c37899ae0e2e0#A#compaction#303 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:57:48,313 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/df0ba5c4dcfa4f19ac13577db8d10007 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/df0ba5c4dcfa4f19ac13577db8d10007 2023-04-05 04:57:48,317 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 5708f0f53456b130f41c37899ae0e2e0/B of 5708f0f53456b130f41c37899ae0e2e0 into df0ba5c4dcfa4f19ac13577db8d10007(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:57:48,318 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5708f0f53456b130f41c37899ae0e2e0: 2023-04-05 04:57:48,318 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0., storeName=5708f0f53456b130f41c37899ae0e2e0/B, priority=11, startTime=1680670668256; duration=0sec 2023-04-05 04:57:48,318 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:57:48,318 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 04:57:48,320 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57086 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 04:57:48,320 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): 5708f0f53456b130f41c37899ae0e2e0/C is initiating minor compaction (all files) 2023-04-05 04:57:48,320 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5708f0f53456b130f41c37899ae0e2e0/C in TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:48,321 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/0cac5d39573c4344b6718fcb29b77c34, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/6a559d5bfa4f4a949f4463fe00bb09b3, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/533eca10fbbe42e39d786d2fd1ff00c1, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/adbf54ec6fdb45acbd92fdb0ad316369, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/75e7b6ff42ab4113b4e2e6205999bb40] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp, totalSize=55.7 K 2023-04-05 04:57:48,321 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 0cac5d39573c4344b6718fcb29b77c34, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680670661538 2023-04-05 04:57:48,321 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 6a559d5bfa4f4a949f4463fe00bb09b3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680670661575 2023-04-05 04:57:48,322 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 533eca10fbbe42e39d786d2fd1ff00c1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680670663701 2023-04-05 04:57:48,322 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting adbf54ec6fdb45acbd92fdb0ad316369, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1680670664388 2023-04-05 04:57:48,323 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 75e7b6ff42ab4113b4e2e6205999bb40, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680670665756 2023-04-05 04:57:48,332 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): 5708f0f53456b130f41c37899ae0e2e0#C#compaction#305 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:57:48,350 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/b99cd8ec1571482a8c643e3a298d548e as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/b99cd8ec1571482a8c643e3a298d548e 2023-04-05 04:57:48,363 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 5708f0f53456b130f41c37899ae0e2e0/C of 5708f0f53456b130f41c37899ae0e2e0 into b99cd8ec1571482a8c643e3a298d548e(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:57:48,363 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5708f0f53456b130f41c37899ae0e2e0: 2023-04-05 04:57:48,363 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0., storeName=5708f0f53456b130f41c37899ae0e2e0/C, priority=11, startTime=1680670668256; duration=0sec 2023-04-05 04:57:48,363 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:57:48,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:57:48,558 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:48,622 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:48,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54608 deadline: 1680670728614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,731 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:48,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54608 deadline: 1680670728723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,739 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304051ca91d51f2724604957a459684c55944 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304051ca91d51f2724604957a459684c55944 2023-04-05 04:57:48,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:48,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54542 deadline: 1680670728824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,838 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:48,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54552 deadline: 1680670728829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,844 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:48,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54584 deadline: 1680670728837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:48,944 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:48,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54608 deadline: 1680670728938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,149 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/9108dcdb11a24828bc62023ddd9cbdee as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/9108dcdb11a24828bc62023ddd9cbdee 2023-04-05 04:57:49,155 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 5708f0f53456b130f41c37899ae0e2e0/A of 5708f0f53456b130f41c37899ae0e2e0 into 9108dcdb11a24828bc62023ddd9cbdee(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:57:49,155 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5708f0f53456b130f41c37899ae0e2e0: 2023-04-05 04:57:49,155 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0., storeName=5708f0f53456b130f41c37899ae0e2e0/A, priority=11, startTime=1680670668256; duration=0sec 2023-04-05 04:57:49,155 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:57:49,159 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=143, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/6144753bd6004cf290290ef77a09e2b3 2023-04-05 04:57:49,195 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/dbdb8dc4e0cd46ceb6f88d2c495e9fba 2023-04-05 04:57:49,251 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:49,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54608 deadline: 1680670729247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,616 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/a39e6337dbf743799871a1ff705cea3b 2023-04-05 04:57:49,620 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/6144753bd6004cf290290ef77a09e2b3 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/6144753bd6004cf290290ef77a09e2b3 2023-04-05 04:57:49,623 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/6144753bd6004cf290290ef77a09e2b3, entries=150, sequenceid=143, filesize=25.3 K 2023-04-05 04:57:49,624 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/dbdb8dc4e0cd46ceb6f88d2c495e9fba as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/dbdb8dc4e0cd46ceb6f88d2c495e9fba 2023-04-05 04:57:49,636 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/dbdb8dc4e0cd46ceb6f88d2c495e9fba, entries=150, sequenceid=143, filesize=11.8 K 2023-04-05 04:57:49,637 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/a39e6337dbf743799871a1ff705cea3b as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/a39e6337dbf743799871a1ff705cea3b 2023-04-05 04:57:49,648 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/a39e6337dbf743799871a1ff705cea3b, entries=150, sequenceid=143, filesize=11.8 K 2023-04-05 04:57:49,650 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 5708f0f53456b130f41c37899ae0e2e0 in 1376ms, sequenceid=143, compaction requested=false 2023-04-05 04:57:49,650 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5708f0f53456b130f41c37899ae0e2e0: 2023-04-05 04:57:49,650 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:49,650 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:49,650 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:49,650 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,650 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:49,650 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:49,651 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,651 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:49,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:49,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:49,652 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:49,652 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:49,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:49,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:49,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:49,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:49,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:49,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ca1291[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:49,653 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:49,653 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:49,654 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:49,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:49,654 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,654 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:49,654 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:49,654 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:49,655 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:49,657 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:49,657 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:49,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:49,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:49,657 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:49,657 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:49,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:49,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:49,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:49,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:49,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:49,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:49,659 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:49,659 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:49,659 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:49,659 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:49,659 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:49,668 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:49,668 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:49,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:49,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:49,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:49,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:49,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:49,668 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:49,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:49,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:49,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:49,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:49,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:49,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:49,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:49,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,670 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:49,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:49,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,671 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,680 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,680 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:49,680 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:49,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:49,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:49,680 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:49,680 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:49,680 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,680 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:49,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:49,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:49,680 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:49,681 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:49,681 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:49,681 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:49,681 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:49,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:49,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:49,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:57:49,759 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5708f0f53456b130f41c37899ae0e2e0 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 04:57:49,759 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=A 2023-04-05 04:57:49,759 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:49,759 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=B 2023-04-05 04:57:49,759 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:49,759 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=C 2023-04-05 04:57:49,759 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:49,781 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:49,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:49,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:49,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:49,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:49,787 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:49,787 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:49,787 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:49,787 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:49,788 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:49,788 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:49,788 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,788 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:49,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:49,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:49,789 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,789 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:49,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:49,790 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:49,790 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:49,790 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:49,790 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:49,790 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:49,790 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:49,809 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:49,809 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:49,809 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:49,810 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:49,810 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:49,810 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:49,810 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:49,810 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:49,810 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,810 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:49,810 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:49,811 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,811 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:49,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:49,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:49,811 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:49,811 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:49,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:49,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:49,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:49,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:49,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:49,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@350dfee[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:49,813 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:49,813 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:49,813 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:49,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:49,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:49,813 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:49,814 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:49,814 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:49,814 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,814 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:49,814 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:49,814 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:49,814 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:49,814 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:49,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:49,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:49,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:49,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:49,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:49,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:49,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:49,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:49,817 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:49,817 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:49,817 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:49,817 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:49,817 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:49,818 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:49,818 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:49,818 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:49,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:49,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:49,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:49,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:49,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:49,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:49,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:49,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:49,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:49,822 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:49,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:49,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:49,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,823 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:49,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:49,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,825 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:49,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:49,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:49,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:49,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:49,825 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:49,825 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:49,825 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:49,826 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:49,826 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:49,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:49,826 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:49,826 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:49,826 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,826 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:49,826 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:49,826 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:49,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670669830, max timestamp=1680670669884Num uniques 859; 2023-04-05 04:57:49,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:57:49,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670669830, max timestamp=1680670669884Num uniques 859; 2023-04-05 04:57:49,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670669830, max timestamp=1680670669884Num uniques 859; 2023-04-05 04:57:49,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:57:49,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:57:49,926 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:49,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:49,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:49,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:49,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:49,929 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:49,929 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:49,930 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:49,930 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:49,930 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:49,930 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:49,931 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,931 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:49,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:49,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:49,931 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,931 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:49,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:49,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:49,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:49,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:49,932 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:49,934 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:49,934 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:49,934 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:49,934 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:49,934 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:49,936 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:49,936 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:49,936 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:49,936 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:49,936 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:49,936 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:49,936 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:49,936 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:49,936 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,936 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:49,936 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:49,937 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:49,938 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:49,938 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:49,942 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:49,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:49,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:49,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:49,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:49,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:49,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:49,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47a0fe4f[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:49,944 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:49,944 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:49,951 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:49,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:49,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:49,952 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:49,952 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,952 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:49,952 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:49,952 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:49,955 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,955 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:49,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:49,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:49,955 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:49,955 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:49,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:49,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:49,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:49,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:49,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:49,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:49,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:49,958 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:49,958 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:49,958 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:49,958 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:49,958 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:49,959 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:49,959 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:49,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:49,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:49,959 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:49,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:49,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:49,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:49,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:49,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:49,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:49,960 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:49,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:49,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:49,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:49,961 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:49,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:49,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,963 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:49,963 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:49,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:49,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:49,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:49,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:49,964 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:49,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:49,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:49,964 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,964 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:49,964 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:49,964 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,964 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:49,965 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:49,965 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:49,965 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:49,965 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:49,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:49,970 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:49,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54608 deadline: 1680670729957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:49,983 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:49,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670729970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,065 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:50,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:50,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:50,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:50,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:50,071 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:50,071 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:50,071 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:50,071 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:50,073 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:50,073 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:50,074 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:50,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:50,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:50,075 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:50,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:50,076 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,076 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:50,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:50,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:50,076 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:50,078 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:50,078 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:50,078 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:50,078 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:50,078 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:50,079 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:50,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54608 deadline: 1680670730071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,087 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:50,087 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:50,087 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:50,087 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:50,087 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:50,087 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:50,087 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:50,087 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:50,087 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,087 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:50,087 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:50,091 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:50,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670730085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,098 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:50,098 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:50,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:50,098 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:50,098 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:50,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:50,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:50,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:50,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:50,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:50,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22c77e17[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:50,109 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:50,109 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:50,110 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:50,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:50,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:50,110 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:50,110 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:50,110 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:50,113 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,113 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:50,114 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,114 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:50,115 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:50,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:50,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:50,115 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:50,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:50,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:50,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:50,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:50,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:50,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:50,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:50,127 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:50,127 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:50,127 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:50,127 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:50,127 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:50,128 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:50,128 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:50,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:50,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:50,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:50,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:50,129 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:50,130 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,130 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:50,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:50,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:50,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:50,131 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:50,131 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:50,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:50,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:50,132 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,132 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:50,132 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:50,132 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,132 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:50,132 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:50,132 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:50,132 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:50,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:50,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:50,132 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:50,132 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:50,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:50,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:50,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:50,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:50,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:50,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:50,179 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405a3f87894e56d48448f53ae81ced1e2c6 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405a3f87894e56d48448f53ae81ced1e2c6 2023-04-05 04:57:50,195 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=157, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/d7956de298824282b9fc6907ce674152 2023-04-05 04:57:50,209 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/65ff6bca970e4b508418c71892c40861 2023-04-05 04:57:50,232 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:50,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:50,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:50,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:50,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:50,236 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:50,236 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:50,236 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:50,236 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:50,237 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:50,237 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:50,238 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:50,238 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:50,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:50,238 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,238 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:50,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:50,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:50,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:50,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:50,239 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:50,239 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:50,239 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:50,239 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:50,239 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:50,239 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:50,239 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:50,239 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:50,239 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:50,239 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:50,240 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:50,240 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:50,240 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:50,240 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:50,240 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,240 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:50,240 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:50,241 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,241 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:50,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:50,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:50,241 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:50,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:50,241 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:50,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:50,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:50,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:50,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:50,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72c9ae44[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:50,242 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:50,242 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:50,243 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:50,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:50,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:50,243 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:50,243 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:50,243 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:50,245 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:50,245 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:50,245 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:50,245 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,245 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:50,245 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:50,245 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:50,245 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:50,246 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:50,246 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:50,246 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:50,252 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:50,252 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:50,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:50,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:50,252 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:50,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:50,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:50,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:50,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:50,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:50,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:50,252 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:50,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:50,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:50,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:50,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,253 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:50,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:50,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,255 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,255 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:50,255 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:50,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:50,255 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:50,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:50,255 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,256 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:50,256 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:50,256 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:50,255 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:50,256 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:50,256 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:50,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:50,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:50,255 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:50,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:50,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:50,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:50,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54608 deadline: 1680670730281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,298 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:50,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670730293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,356 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:50,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:50,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:50,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:50,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:50,373 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:50,373 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:50,373 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:50,374 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:50,374 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:50,374 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:50,375 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:50,375 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:50,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:50,375 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,375 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:50,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:50,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:50,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:50,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:50,376 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:50,386 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:50,386 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:50,386 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:50,386 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:50,386 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:50,388 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:50,388 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:50,388 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:50,388 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:50,388 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:50,388 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:50,388 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:50,388 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:50,388 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,388 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:50,388 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:50,390 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,390 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:50,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:50,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:50,390 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:50,390 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:50,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:50,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:50,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:50,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:50,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:50,391 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:50,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7763b05c[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:50,391 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:50,392 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:50,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:50,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:50,392 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,392 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:50,392 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:50,393 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:50,393 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:50,393 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:50,393 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:50,394 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:50,394 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:50,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:50,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:50,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:50,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:50,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:50,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:50,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:50,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:50,396 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:50,396 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:50,396 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:50,396 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:50,396 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:50,397 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:50,397 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:50,397 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:50,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:50,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:50,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:50,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:50,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:50,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:50,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:50,398 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:50,399 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:50,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:50,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,401 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,401 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:50,401 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:50,401 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,401 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:50,401 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:50,402 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:50,403 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:50,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:50,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:50,403 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:50,403 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:50,403 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:50,403 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:50,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:50,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:50,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:50,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:50,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:50,503 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:50,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:50,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:50,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:50,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:50,510 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:50,510 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:50,510 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:50,510 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:50,510 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:50,510 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:50,511 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,511 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,511 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:50,512 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:50,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:50,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:50,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:50,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:50,513 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:50,526 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:50,526 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:50,526 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:50,526 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:50,526 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:50,527 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:50,527 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:50,527 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:50,527 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:50,528 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:50,528 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:50,528 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:50,528 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:50,528 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,528 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:50,528 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:50,529 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:50,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:50,529 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:50,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:50,529 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:50,529 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:50,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:50,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:50,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:50,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:50,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37717a79[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:50,532 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:50,532 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:50,532 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:50,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:50,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:50,532 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:50,532 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:50,532 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:50,533 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,533 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:50,533 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,533 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:50,533 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:50,533 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:50,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:50,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:50,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:50,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:50,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:50,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:50,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:50,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:50,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:50,537 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:50,537 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:50,537 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:50,537 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:50,537 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:50,538 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:50,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:50,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:50,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:50,538 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:50,538 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:50,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:50,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:50,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:50,538 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:50,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:50,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:50,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:50,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:50,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:50,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,539 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:50,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:50,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,541 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:50,541 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:50,541 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:50,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:50,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:50,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:50,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:50,541 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:50,541 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:50,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:50,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:50,542 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,542 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:50,542 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:50,542 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,542 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:50,542 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:50,542 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:50,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:50,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54608 deadline: 1680670730589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,608 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:50,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670730600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,620 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/759fdd54a2414a189db708d2040da14b 2023-04-05 04:57:50,623 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/d7956de298824282b9fc6907ce674152 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/d7956de298824282b9fc6907ce674152 2023-04-05 04:57:50,628 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/d7956de298824282b9fc6907ce674152, entries=200, sequenceid=157, filesize=32.1 K 2023-04-05 04:57:50,629 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/65ff6bca970e4b508418c71892c40861 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/65ff6bca970e4b508418c71892c40861 2023-04-05 04:57:50,634 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/65ff6bca970e4b508418c71892c40861, entries=150, sequenceid=157, filesize=11.8 K 2023-04-05 04:57:50,635 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/759fdd54a2414a189db708d2040da14b as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/759fdd54a2414a189db708d2040da14b 2023-04-05 04:57:50,638 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/759fdd54a2414a189db708d2040da14b, entries=150, sequenceid=157, filesize=11.8 K 2023-04-05 04:57:50,639 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 5708f0f53456b130f41c37899ae0e2e0 in 880ms, sequenceid=157, compaction requested=true 2023-04-05 04:57:50,639 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5708f0f53456b130f41c37899ae0e2e0: 2023-04-05 04:57:50,639 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:57:50,639 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:57:50,639 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:57:50,639 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:57:50,640 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:57:50,640 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36130 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:57:50,640 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84700 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:57:50,640 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): 5708f0f53456b130f41c37899ae0e2e0/B is initiating minor compaction (all files) 2023-04-05 04:57:50,640 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): 5708f0f53456b130f41c37899ae0e2e0/A is initiating minor compaction (all files) 2023-04-05 04:57:50,640 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5708f0f53456b130f41c37899ae0e2e0/B in TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:50,640 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5708f0f53456b130f41c37899ae0e2e0/A in TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:50,640 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/df0ba5c4dcfa4f19ac13577db8d10007, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/dbdb8dc4e0cd46ceb6f88d2c495e9fba, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/65ff6bca970e4b508418c71892c40861] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp, totalSize=35.3 K 2023-04-05 04:57:50,640 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/9108dcdb11a24828bc62023ddd9cbdee, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/6144753bd6004cf290290ef77a09e2b3, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/d7956de298824282b9fc6907ce674152] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp, totalSize=82.7 K 2023-04-05 04:57:50,641 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting df0ba5c4dcfa4f19ac13577db8d10007, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680670665756 2023-04-05 04:57:50,641 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 9108dcdb11a24828bc62023ddd9cbdee, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680670665756 2023-04-05 04:57:50,641 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting dbdb8dc4e0cd46ceb6f88d2c495e9fba, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1680670666658 2023-04-05 04:57:50,641 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:50,641 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 6144753bd6004cf290290ef77a09e2b3, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1680670666658 2023-04-05 04:57:50,642 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 65ff6bca970e4b508418c71892c40861, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1680670668605 2023-04-05 04:57:50,642 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting d7956de298824282b9fc6907ce674152, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1680670668580 2023-04-05 04:57:50,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:50,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:50,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:50,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:50,658 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:50,659 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:50,659 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:50,659 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:50,659 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:50,659 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:50,660 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,660 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:50,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:50,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:50,660 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,660 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:50,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:50,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:50,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:50,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:50,661 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:50,663 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): 5708f0f53456b130f41c37899ae0e2e0#B#compaction#311 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:57:50,670 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:50,670 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:50,670 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:50,670 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:50,670 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:50,682 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:50,682 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:50,682 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:50,682 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:50,682 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5708f0f53456b130f41c37899ae0e2e0 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-05 04:57:50,682 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=A 2023-04-05 04:57:50,683 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:50,683 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=B 2023-04-05 04:57:50,683 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:50,683 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=C 2023-04-05 04:57:50,683 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:50,718 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5708f0f53456b130f41c37899ae0e2e0#A#compaction#312 average throughput is 0.49 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:57:50,725 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/dcdeac5064d344b6a7c01eb2997a4a73 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/dcdeac5064d344b6a7c01eb2997a4a73 2023-04-05 04:57:50,729 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5708f0f53456b130f41c37899ae0e2e0/B of 5708f0f53456b130f41c37899ae0e2e0 into dcdeac5064d344b6a7c01eb2997a4a73(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:57:50,729 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5708f0f53456b130f41c37899ae0e2e0: 2023-04-05 04:57:50,730 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0., storeName=5708f0f53456b130f41c37899ae0e2e0/B, priority=13, startTime=1680670670639; duration=0sec 2023-04-05 04:57:50,730 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:57:50,730 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:57:50,731 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36130 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:57:50,731 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): 5708f0f53456b130f41c37899ae0e2e0/C is initiating minor compaction (all files) 2023-04-05 04:57:50,731 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5708f0f53456b130f41c37899ae0e2e0/C in TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:50,731 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/b99cd8ec1571482a8c643e3a298d548e, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/a39e6337dbf743799871a1ff705cea3b, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/759fdd54a2414a189db708d2040da14b] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp, totalSize=35.3 K 2023-04-05 04:57:50,731 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting b99cd8ec1571482a8c643e3a298d548e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680670665756 2023-04-05 04:57:50,732 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting a39e6337dbf743799871a1ff705cea3b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1680670666658 2023-04-05 04:57:50,732 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 759fdd54a2414a189db708d2040da14b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1680670668605 2023-04-05 04:57:50,743 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): 5708f0f53456b130f41c37899ae0e2e0#C#compaction#314 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:57:50,766 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304051cce739474824c288a10d3b8f754f02d to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304051cce739474824c288a10d3b8f754f02d 2023-04-05 04:57:50,772 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/aea24ad35b8c45369808515755e0280a as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/aea24ad35b8c45369808515755e0280a 2023-04-05 04:57:50,778 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5708f0f53456b130f41c37899ae0e2e0/C of 5708f0f53456b130f41c37899ae0e2e0 into aea24ad35b8c45369808515755e0280a(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:57:50,778 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5708f0f53456b130f41c37899ae0e2e0: 2023-04-05 04:57:50,778 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0., storeName=5708f0f53456b130f41c37899ae0e2e0/C, priority=13, startTime=1680670670640; duration=0sec 2023-04-05 04:57:50,778 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:57:50,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:57:50,847 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:50,880 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:50,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54584 deadline: 1680670730872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,881 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:50,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54552 deadline: 1680670730872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,881 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:50,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54542 deadline: 1680670730872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,984 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:50,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54584 deadline: 1680670730981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:50,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54552 deadline: 1680670730982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:50,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:50,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54542 deadline: 1680670730982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:51,110 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:51,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54608 deadline: 1680670731103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:51,119 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:51,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670731111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:51,189 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=193, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/82fdae4e85814080b564d772f54f322f 2023-04-05 04:57:51,190 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:51,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54552 deadline: 1680670731186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:51,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:51,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54542 deadline: 1680670731186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:51,198 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:51,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54584 deadline: 1680670731191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:51,208 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/73649c85cb3f44de9fc3d8931b201f48 2023-04-05 04:57:51,496 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:51,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54552 deadline: 1680670731494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:51,507 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:51,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54542 deadline: 1680670731499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:51,507 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:51,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54584 deadline: 1680670731501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:51,563 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/120261db92b2459bb056731a4b4d876c as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/120261db92b2459bb056731a4b4d876c 2023-04-05 04:57:51,566 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5708f0f53456b130f41c37899ae0e2e0/A of 5708f0f53456b130f41c37899ae0e2e0 into 120261db92b2459bb056731a4b4d876c(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:57:51,567 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5708f0f53456b130f41c37899ae0e2e0: 2023-04-05 04:57:51,567 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0., storeName=5708f0f53456b130f41c37899ae0e2e0/A, priority=13, startTime=1680670670639; duration=0sec 2023-04-05 04:57:51,567 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:57:51,630 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/749447674c724ffc9b84e40292912534 2023-04-05 04:57:51,634 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/82fdae4e85814080b564d772f54f322f as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/82fdae4e85814080b564d772f54f322f 2023-04-05 04:57:51,636 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/82fdae4e85814080b564d772f54f322f, entries=150, sequenceid=193, filesize=25.3 K 2023-04-05 04:57:51,637 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/73649c85cb3f44de9fc3d8931b201f48 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/73649c85cb3f44de9fc3d8931b201f48 2023-04-05 04:57:51,640 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/73649c85cb3f44de9fc3d8931b201f48, entries=150, sequenceid=193, filesize=11.8 K 2023-04-05 04:57:51,640 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/749447674c724ffc9b84e40292912534 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/749447674c724ffc9b84e40292912534 2023-04-05 04:57:51,644 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/749447674c724ffc9b84e40292912534, entries=150, sequenceid=193, filesize=11.8 K 2023-04-05 04:57:51,644 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 5708f0f53456b130f41c37899ae0e2e0 in 962ms, sequenceid=193, compaction requested=false 2023-04-05 04:57:51,644 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5708f0f53456b130f41c37899ae0e2e0: 2023-04-05 04:57:51,645 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:51,645 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:51,645 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:51,647 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:51,648 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:51,648 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:51,648 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:51,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:51,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:51,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:51,649 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:51,649 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:51,649 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:51,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:51,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:51,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:51,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:51,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:51,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:51,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2edf9d46[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:51,651 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:51,651 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:51,652 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:51,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:51,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:51,652 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:51,652 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:51,652 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:51,652 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:51,652 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:51,653 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:51,653 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:51,653 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:51,653 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:51,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:51,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:51,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:51,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:51,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:51,655 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:51,655 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:51,655 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:51,655 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:51,655 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:51,657 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:51,657 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:51,657 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:51,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:51,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:51,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:51,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:51,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:51,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:51,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:51,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:51,657 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:51,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:51,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:51,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:51,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:51,658 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:51,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:51,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:51,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:51,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:51,659 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:51,659 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:51,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:51,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:51,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:51,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:51,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:51,660 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:51,660 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:51,660 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:51,660 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:51,660 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:51,660 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:51,660 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:51,661 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:51,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:51,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:51,661 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:51,661 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:51,761 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:51,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:51,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:51,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:51,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:51,764 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:51,764 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:51,764 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:51,764 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:51,764 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:51,764 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:51,765 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:51,765 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:51,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:51,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:51,765 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:51,765 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:51,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:51,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:51,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:51,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:51,766 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:51,769 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:51,769 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:51,769 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:51,769 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:51,769 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:51,769 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:51,769 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:51,769 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:51,769 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:51,769 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5708f0f53456b130f41c37899ae0e2e0 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 04:57:51,769 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=A 2023-04-05 04:57:51,769 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:51,769 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=B 2023-04-05 04:57:51,769 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:51,770 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=C 2023-04-05 04:57:51,770 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:51,793 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405556aba58165f40bfbe1cafa95aa6aed7 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405556aba58165f40bfbe1cafa95aa6aed7 2023-04-05 04:57:52,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:57:52,000 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:52,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680670672022, max timestamp=1680670672044Num uniques 859; 2023-04-05 04:57:52,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680670672022, max timestamp=1680670672044Num uniques 859; 2023-04-05 04:57:52,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680670672022, max timestamp=1680670672044Num uniques 859; 2023-04-05 04:57:52,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 04:57:52,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 04:57:52,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 04:57:52,130 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:52,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54542 deadline: 1680670732123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:52,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54552 deadline: 1680670732124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:52,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:52,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670732126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54584 deadline: 1680670732126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:52,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54608 deadline: 1680670732127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,199 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=207, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/d8e570d7a24b4ffd93e51fce18000d40 2023-04-05 04:57:52,219 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=207 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/e1897cf0d0a6465db040b8a34876a854 2023-04-05 04:57:52,231 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=207 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/6c55c83d84d14ce194684ce3736bb1cb 2023-04-05 04:57:52,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:52,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54542 deadline: 1680670732232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,238 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/d8e570d7a24b4ffd93e51fce18000d40 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/d8e570d7a24b4ffd93e51fce18000d40 2023-04-05 04:57:52,241 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/d8e570d7a24b4ffd93e51fce18000d40, entries=150, sequenceid=207, filesize=25.3 K 2023-04-05 04:57:52,241 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/e1897cf0d0a6465db040b8a34876a854 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/e1897cf0d0a6465db040b8a34876a854 2023-04-05 04:57:52,244 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/e1897cf0d0a6465db040b8a34876a854, entries=150, sequenceid=207, filesize=11.8 K 2023-04-05 04:57:52,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:52,245 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/6c55c83d84d14ce194684ce3736bb1cb as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/6c55c83d84d14ce194684ce3736bb1cb 2023-04-05 04:57:52,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54552 deadline: 1680670732240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:52,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54584 deadline: 1680670732241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:52,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54608 deadline: 1680670732241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,251 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/6c55c83d84d14ce194684ce3736bb1cb, entries=150, sequenceid=207, filesize=11.8 K 2023-04-05 04:57:52,254 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 5708f0f53456b130f41c37899ae0e2e0 in 485ms, sequenceid=207, compaction requested=true 2023-04-05 04:57:52,254 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5708f0f53456b130f41c37899ae0e2e0: 2023-04-05 04:57:52,254 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:52,254 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:52,254 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:52,254 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,254 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:52,254 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:52,258 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:52,258 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:52,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:52,258 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:52,258 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:52,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:52,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:52,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:52,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:52,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:52,259 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:52,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11dbad57[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:52,259 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:52,259 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:52,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:52,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:52,260 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:52,260 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:52,260 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:52,260 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,260 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:52,260 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:52,260 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:52,260 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,260 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:52,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:52,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:52,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:52,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:52,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:52,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:52,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:52,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:52,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:52,262 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:52,262 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:52,262 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:52,262 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:52,262 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:52,263 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:52,263 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:52,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:52,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:52,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:52,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:52,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:52,263 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:52,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:52,264 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:52,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:52,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:52,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:52,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:52,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:52,265 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:52,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:52,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,266 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,266 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:52,266 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:52,266 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,266 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:52,266 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:52,266 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:52,266 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:52,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:52,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:52,266 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:52,266 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:52,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:52,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:52,266 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:52,267 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:52,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:52,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:52,367 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:52,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:52,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:52,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:52,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:52,369 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:52,369 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:52,369 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:52,370 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:52,370 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:52,371 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:52,371 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:52,371 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:52,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:52,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:52,372 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,372 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:52,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:52,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:52,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:52,372 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:52,372 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:52,372 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:52,372 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:52,372 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:52,372 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:52,373 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:52,373 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:52,373 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:52,373 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:52,373 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5708f0f53456b130f41c37899ae0e2e0 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-05 04:57:52,374 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=A 2023-04-05 04:57:52,374 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:52,374 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=B 2023-04-05 04:57:52,374 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:52,374 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=C 2023-04-05 04:57:52,374 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:52,394 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304056eb9e87a2af54c3aa2586c4e88c1aa0f to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304056eb9e87a2af54c3aa2586c4e88c1aa0f 2023-04-05 04:57:52,428 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=241, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/5b8960c11f32419ab53052f0d9badc4b 2023-04-05 04:57:52,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:57:52,444 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:52,462 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=241 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/24d9708f1b0149b98f9e097bb4154932 2023-04-05 04:57:52,469 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:52,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54608 deadline: 1680670732455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,473 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:52,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54542 deadline: 1680670732461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,474 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:52,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54552 deadline: 1680670732468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,476 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:52,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54584 deadline: 1680670732469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:52,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54608 deadline: 1680670732570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,580 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:52,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54542 deadline: 1680670732574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,581 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:52,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54552 deadline: 1680670732575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,581 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:52,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54584 deadline: 1680670732577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:52,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54608 deadline: 1680670732777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,788 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:52,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54542 deadline: 1680670732781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,788 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:52,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54552 deadline: 1680670732781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:52,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54584 deadline: 1680670732783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,872 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=241 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/6fea3ce1ff1c4ad8a334b125f8c98fbb 2023-04-05 04:57:52,879 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/5b8960c11f32419ab53052f0d9badc4b as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/5b8960c11f32419ab53052f0d9badc4b 2023-04-05 04:57:52,882 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/5b8960c11f32419ab53052f0d9badc4b, entries=150, sequenceid=241, filesize=25.3 K 2023-04-05 04:57:52,883 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/24d9708f1b0149b98f9e097bb4154932 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/24d9708f1b0149b98f9e097bb4154932 2023-04-05 04:57:52,887 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/24d9708f1b0149b98f9e097bb4154932, entries=150, sequenceid=241, filesize=11.8 K 2023-04-05 04:57:52,888 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/6fea3ce1ff1c4ad8a334b125f8c98fbb as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/6fea3ce1ff1c4ad8a334b125f8c98fbb 2023-04-05 04:57:52,891 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/6fea3ce1ff1c4ad8a334b125f8c98fbb, entries=150, sequenceid=241, filesize=11.8 K 2023-04-05 04:57:52,893 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for 5708f0f53456b130f41c37899ae0e2e0 in 520ms, sequenceid=241, compaction requested=true 2023-04-05 04:57:52,893 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5708f0f53456b130f41c37899ae0e2e0: 2023-04-05 04:57:52,893 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:52,893 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:52,893 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:52,893 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,893 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:52,893 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:52,894 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:52,894 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:52,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:52,894 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:52,894 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:52,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:52,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:52,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:52,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:52,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:52,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bd00e60[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:52,895 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:52,895 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:52,897 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:52,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:52,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:52,897 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:52,897 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,897 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:52,897 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:52,897 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:52,898 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,898 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:52,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:52,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:52,898 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:52,899 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:52,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:52,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:52,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:52,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:52,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:52,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:52,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:52,900 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:52,900 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:52,900 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:52,900 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:52,900 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:52,901 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:52,901 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:52,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:52,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:52,902 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:52,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:52,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:52,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:52,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:52,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:52,902 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:52,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:52,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:52,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:52,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:52,904 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:52,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:52,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,905 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,905 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:52,905 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:52,905 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:52,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:52,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:52,905 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:52,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:52,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:52,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:52,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:52,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:52,905 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:52,905 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:52,906 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:52,906 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:52,906 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:52,906 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:52,906 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:53,006 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:53,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:53,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:53,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:53,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:53,008 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:53,008 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:53,008 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:53,008 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:53,009 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:53,009 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:53,010 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:53,010 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:53,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:53,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:53,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:53,011 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:53,011 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:53,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:53,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:53,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:53,011 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:53,011 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:53,011 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:53,012 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:53,012 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:53,012 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:53,012 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:53,012 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:53,012 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:53,012 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:53,012 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5708f0f53456b130f41c37899ae0e2e0 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 04:57:53,012 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=A 2023-04-05 04:57:53,012 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:53,012 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=B 2023-04-05 04:57:53,012 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:53,013 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=C 2023-04-05 04:57:53,013 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:53,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:57:53,088 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:53,121 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:53,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54552 deadline: 1680670733115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:53,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670673100, max timestamp=1680670673120Num uniques 859; 2023-04-05 04:57:53,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670673100, max timestamp=1680670673120Num uniques 859; 2023-04-05 04:57:53,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:57:53,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:57:53,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670673100, max timestamp=1680670673120Num uniques 859; 2023-04-05 04:57:53,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:57:53,172 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:53,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54608 deadline: 1680670733160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:53,177 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:53,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54542 deadline: 1680670733171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:53,177 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:53,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54584 deadline: 1680670733171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:53,229 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:53,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54552 deadline: 1680670733222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:53,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:53,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54608 deadline: 1680670733273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:53,287 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:53,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54542 deadline: 1680670733278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:53,288 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:53,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54584 deadline: 1680670733279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:53,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:53,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54552 deadline: 1680670733431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:53,447 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405c42e2a67575047ef89a927abc3859cdd to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405c42e2a67575047ef89a927abc3859cdd 2023-04-05 04:57:53,466 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=252, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/8091f18f87c946fe9058db4c1abec559 2023-04-05 04:57:53,487 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:53,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54608 deadline: 1680670733481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:53,494 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:53,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54542 deadline: 1680670733489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:53,494 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:53,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54584 deadline: 1680670733489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:53,745 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:53,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54552 deadline: 1680670733741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:53,794 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:53,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54608 deadline: 1680670733790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:53,802 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:53,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54584 deadline: 1680670733796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:53,802 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:53,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54542 deadline: 1680670733797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:53,875 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=252 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/22ebd28e1e4941218bdbf04b5c91ed77 2023-04-05 04:57:53,888 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=252 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/7da440b195724986b95c74e2da4ee3d7 2023-04-05 04:57:53,892 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/8091f18f87c946fe9058db4c1abec559 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/8091f18f87c946fe9058db4c1abec559 2023-04-05 04:57:53,898 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/8091f18f87c946fe9058db4c1abec559, entries=150, sequenceid=252, filesize=25.3 K 2023-04-05 04:57:53,898 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/22ebd28e1e4941218bdbf04b5c91ed77 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/22ebd28e1e4941218bdbf04b5c91ed77 2023-04-05 04:57:53,903 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/22ebd28e1e4941218bdbf04b5c91ed77, entries=150, sequenceid=252, filesize=11.8 K 2023-04-05 04:57:53,904 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/7da440b195724986b95c74e2da4ee3d7 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/7da440b195724986b95c74e2da4ee3d7 2023-04-05 04:57:53,907 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/7da440b195724986b95c74e2da4ee3d7, entries=150, sequenceid=252, filesize=11.8 K 2023-04-05 04:57:53,908 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 5708f0f53456b130f41c37899ae0e2e0 in 896ms, sequenceid=252, compaction requested=true 2023-04-05 04:57:53,908 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5708f0f53456b130f41c37899ae0e2e0: 2023-04-05 04:57:53,908 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:53,908 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:53,908 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:53,908 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:53,908 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:53,908 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:53,909 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:53,909 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:53,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:53,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:53,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:53,909 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:53,909 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:53,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:53,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:53,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:53,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:53,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:53,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:53,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@474e11d1[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:53,910 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:53,910 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:53,911 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:53,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:53,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:53,911 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:53,911 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:53,911 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:53,911 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:53,911 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:53,912 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:53,912 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:53,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:53,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:53,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:53,912 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:53,912 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:53,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:53,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:53,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:53,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:53,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:53,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:53,913 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:53,913 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:53,913 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:53,913 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:53,913 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:53,914 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:53,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:53,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:53,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:53,914 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:53,914 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:53,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:53,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:53,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:53,914 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:53,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:53,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:53,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:53,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:53,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:53,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:53,915 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:53,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:53,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:53,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:53,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:53,916 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:53,916 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:53,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:53,916 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:53,916 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:53,917 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:53,917 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:53,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:53,917 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:53,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:53,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:53,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:53,917 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:53,917 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:53,917 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:53,917 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:53,917 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:53,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:53,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:54,017 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:54,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:54,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:54,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:54,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:54,021 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:54,021 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:54,021 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:54,021 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:54,022 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:54,022 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:54,022 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:54,023 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:54,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:54,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:54,023 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:54,023 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:54,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:54,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:54,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:54,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:54,023 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:54,023 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:54,024 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:54,024 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:54,024 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:54,024 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:54,024 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:54,024 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:54,024 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:54,024 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:54,024 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5708f0f53456b130f41c37899ae0e2e0 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-05 04:57:54,024 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=A 2023-04-05 04:57:54,024 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:54,024 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=B 2023-04-05 04:57:54,024 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:54,025 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=C 2023-04-05 04:57:54,025 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:57:54,039 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040507cc19e22e66415083755992b27c8e5e to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040507cc19e22e66415083755992b27c8e5e 2023-04-05 04:57:54,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:57:54,163 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:54,261 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:54,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670734248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:54,263 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:54,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54552 deadline: 1680670734249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:54,306 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:54,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54608 deadline: 1680670734297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:54,308 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:54,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54584 deadline: 1680670734303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:54,317 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:54,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54542 deadline: 1680670734306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:54,374 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:54,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670734362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:54,448 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=287, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/558e8a5b750e4a22aeb14d75282c4414 2023-04-05 04:57:54,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:54,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670734577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:54,858 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=287 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/59285e545fe44dfcaf8bb19b56deb184 2023-04-05 04:57:54,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:54,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670734884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,275 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:55,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54552 deadline: 1680670735269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,282 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=287 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/fc7441fe6c0c4f4dbfef4b2483217522 2023-04-05 04:57:55,289 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/558e8a5b750e4a22aeb14d75282c4414 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/558e8a5b750e4a22aeb14d75282c4414 2023-04-05 04:57:55,295 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/558e8a5b750e4a22aeb14d75282c4414, entries=150, sequenceid=287, filesize=25.4 K 2023-04-05 04:57:55,296 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/59285e545fe44dfcaf8bb19b56deb184 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/59285e545fe44dfcaf8bb19b56deb184 2023-04-05 04:57:55,307 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/59285e545fe44dfcaf8bb19b56deb184, entries=150, sequenceid=287, filesize=11.9 K 2023-04-05 04:57:55,307 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/fc7441fe6c0c4f4dbfef4b2483217522 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/fc7441fe6c0c4f4dbfef4b2483217522 2023-04-05 04:57:55,318 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/fc7441fe6c0c4f4dbfef4b2483217522, entries=150, sequenceid=287, filesize=11.9 K 2023-04-05 04:57:55,319 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 5708f0f53456b130f41c37899ae0e2e0 in 1295ms, sequenceid=287, compaction requested=true 2023-04-05 04:57:55,319 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5708f0f53456b130f41c37899ae0e2e0: 2023-04-05 04:57:55,319 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:55,319 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:55,319 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:55,319 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,319 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:55,319 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:55,320 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:55,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:55,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:55,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:55,321 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:55,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,322 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,322 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:55,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:55,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:55,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15a981d3[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:55,323 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:55,323 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,330 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,330 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:55,330 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:55,330 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:55,330 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,330 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:55,335 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,335 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:55,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:55,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:55,335 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:55,335 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:55,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:55,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:55,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:55,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:55,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:55,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:55,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:55,337 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:55,337 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:55,337 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:55,337 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:55,337 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:55,338 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,338 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:55,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:55,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,338 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:55,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:55,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:55,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:55,349 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:55,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:55,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:55,349 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:55,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:55,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,351 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:55,351 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:55,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:55,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:55,352 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,352 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:55,352 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:55,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:55,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:55,352 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:55,352 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,352 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:55,352 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:55,352 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:55,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:55,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:55,352 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,352 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,352 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:57:55,354 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5708f0f53456b130f41c37899ae0e2e0 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-05 04:57:55,355 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=A 2023-04-05 04:57:55,355 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:55,355 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=B 2023-04-05 04:57:55,355 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:55,355 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=C 2023-04-05 04:57:55,355 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:55,426 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:55,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670735419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,426 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:55,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54584 deadline: 1680670735424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,431 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:55,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54542 deadline: 1680670735425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,433 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:55,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54608 deadline: 1680670735426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,452 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:55,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:55,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:55,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:55,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:55,460 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:55,460 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:55,460 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:55,460 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:55,461 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,461 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:55,461 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:55,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:55,461 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:55,462 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,462 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:55,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:55,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:55,463 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:55,463 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:55,463 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:55,463 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:55,463 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:55,463 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:55,466 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:55,466 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:55,466 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:55,466 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:55,466 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:55,466 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:55,466 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:55,466 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:55,466 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,466 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:55,466 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:55,467 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,467 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:55,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:55,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:55,468 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,468 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:55,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:55,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:55,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:55,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:55,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59543724[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:55,470 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:55,470 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,470 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,470 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:55,470 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:55,471 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:55,471 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,471 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:55,474 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,474 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:55,474 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:55,474 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:55,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:55,476 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:55,476 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:55,476 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:55,476 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:55,476 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:55,477 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,477 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:55,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,477 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:55,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:55,477 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,478 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:55,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,484 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,484 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:55,484 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:55,484 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:55,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:55,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:55,484 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:55,484 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:55,484 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:55,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:55,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:55,484 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:55,485 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,485 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,486 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,486 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:55,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:55,536 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:55,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670735527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,537 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:55,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54584 deadline: 1680670735527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,542 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:55,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54542 deadline: 1680670735532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,542 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:55,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54608 deadline: 1680670735535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,584 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:55,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:55,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:55,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:55,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:55,588 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:55,588 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:55,588 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:55,588 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:55,589 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,589 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:55,589 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,589 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:55,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:55,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:55,590 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,590 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:55,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:55,590 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:55,594 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:55,594 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:55,594 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:55,594 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:55,594 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:55,598 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:55,598 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:55,598 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:55,598 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:55,598 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:55,598 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:55,598 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:55,598 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:55,598 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,598 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:55,598 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:55,601 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,601 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:55,601 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,602 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:55,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:55,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:55,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:55,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:55,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:55,603 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:55,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@220c8160[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:55,603 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,603 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,603 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:55,603 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:55,603 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:55,603 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,603 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:55,604 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,604 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:55,604 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:55,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:55,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:55,604 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:55,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:55,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:55,606 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:55,606 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:55,606 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:55,606 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:55,606 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:55,607 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,607 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:55,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,607 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:55,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:55,607 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:55,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:55,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,608 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:55,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:55,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,609 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:55,609 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,609 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:55,609 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:55,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:55,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:55,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:55,609 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:55,609 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:55,610 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,610 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,610 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:55,610 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:55,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:55,610 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,610 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:55,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:55,710 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:55,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:55,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:55,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:55,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:55,713 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:55,713 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:55,713 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:55,713 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:55,713 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,713 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:55,714 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:55,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:55,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:55,714 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:55,715 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,715 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:55,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:55,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:55,715 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:55,715 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:55,715 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:55,715 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:55,715 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:55,715 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:55,716 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:55,716 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:55,716 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:55,716 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:55,716 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:55,716 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:55,716 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:55,716 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:55,716 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,716 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:55,716 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:55,717 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,717 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:55,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:55,717 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,717 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:55,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:55,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:55,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:55,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:55,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:55,718 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:55,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15fbdc69[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:55,718 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,719 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,719 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:55,719 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:55,719 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:55,720 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,720 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:55,720 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:55,720 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,720 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:55,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:55,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:55,720 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:55,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:55,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:55,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:55,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:55,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:55,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:55,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:55,724 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:55,725 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:55,725 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:55,725 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:55,725 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:55,725 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,725 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,725 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:55,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:55,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:55,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:55,726 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:55,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:55,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:55,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:55,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:55,727 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:55,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:55,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,728 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,728 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:55,728 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:55,728 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:55,728 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:55,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:55,728 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:55,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:55,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:55,728 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:55,728 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,728 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,728 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:55,729 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:55,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:55,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:55,729 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:55,748 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:55,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54584 deadline: 1680670735738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,748 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:55,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670735738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,749 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:55,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54542 deadline: 1680670735744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,749 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:55,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54608 deadline: 1680670735744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,783 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405e93195ed5a9a4c6daa88960fcd2c85c3 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405e93195ed5a9a4c6daa88960fcd2c85c3 2023-04-05 04:57:55,787 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=300, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/6e7d181a6dbf4cd2b2fdad77eb9fb4a9 2023-04-05 04:57:55,829 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:55,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:55,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:55,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:55,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:55,832 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:55,832 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:55,832 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:55,832 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:55,832 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,832 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:55,833 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:55,833 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:55,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:55,833 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,833 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:55,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:55,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:55,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:55,834 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:55,834 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:55,834 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:55,834 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:55,834 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:55,834 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:55,834 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:55,834 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:55,834 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:55,834 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:55,834 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:55,834 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:55,834 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:55,834 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:55,834 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,834 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:55,834 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:55,835 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,835 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:55,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:55,835 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,835 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:55,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:55,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:55,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:55,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:55,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:55,837 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:55,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c85601c[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:55,837 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,837 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,837 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:55,837 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:55,837 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,837 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:55,837 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:55,838 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,838 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:55,838 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:55,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:55,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:55,838 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:55,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:55,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:55,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:55,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:55,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:55,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:55,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:55,840 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:55,840 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:55,840 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:55,840 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:55,840 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:55,841 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,841 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:55,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:55,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,841 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:55,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:55,841 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:55,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:55,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:55,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:55,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:55,842 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:55,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:55,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,844 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,844 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:55,845 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:55,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:55,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:55,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:55,845 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:55,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:55,845 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:55,845 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:55,845 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:55,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:55,845 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,845 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:55,845 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,845 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,845 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:55,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:55,945 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:55,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:55,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:55,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:55,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:55,948 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:55,948 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:55,948 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:55,948 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:55,948 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,948 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:55,949 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:55,949 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:55,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:55,949 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,949 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:55,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:55,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:55,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:55,950 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:55,950 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:55,950 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:55,950 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:55,950 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:55,950 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:55,950 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:55,950 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:55,950 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:55,950 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:55,950 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:55,950 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:55,950 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:55,950 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:55,950 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,950 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:55,950 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:55,951 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,951 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:55,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:55,951 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,951 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:55,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:55,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:55,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:55,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:55,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:55,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@341083bf[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:55,952 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:55,953 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,953 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,953 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:55,953 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,953 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:55,953 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:55,953 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:55,954 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:55,954 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,954 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:55,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:55,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:55,954 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:55,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:55,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:55,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:55,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:55,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:55,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:55,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:55,956 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:55,956 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:55,956 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:55,957 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:55,957 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:55,957 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,957 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:55,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:55,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,957 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:55,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,958 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:55,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:55,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:55,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:55,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:55,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:55,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:55,959 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:55,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:55,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,960 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,960 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:55,960 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:55,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:55,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:55,960 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:55,960 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:55,960 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:55,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:55,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:55,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:55,960 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:55,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:55,960 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:55,960 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,960 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:55,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:55,960 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:55,960 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:56,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:56,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670736049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:56,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54542 deadline: 1680670736049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:56,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54608 deadline: 1680670736051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,057 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:56,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54584 deadline: 1680670736051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,060 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:56,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:56,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:56,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:56,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:56,063 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:56,063 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:56,063 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:56,063 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:56,063 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:56,063 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:56,064 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:56,064 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:56,064 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:56,064 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:56,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:56,065 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:56,065 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:56,065 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:56,065 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:56,065 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:56,065 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:56,065 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:56,065 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:56,065 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:56,065 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:56,065 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:56,065 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:56,065 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:56,065 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:56,065 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,066 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:56,066 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:56,066 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,066 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:56,070 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:56,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:56,070 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:56,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:56,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:56,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:56,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:56,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@725113dc[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:56,071 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:56,071 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:56,076 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:56,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:56,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:56,076 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,077 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:56,076 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:56,077 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:56,077 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:56,077 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:56,077 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,077 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:56,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:56,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:56,077 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:56,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:56,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:56,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:56,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:56,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:56,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:56,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:56,079 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:56,079 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:56,079 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:56,079 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:56,079 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:56,080 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:56,080 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:56,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:56,080 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:56,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:56,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:56,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:56,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:56,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:56,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:56,080 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:56,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:56,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:56,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:56,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:56,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:56,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:56,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,082 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,083 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,083 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:56,083 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:56,083 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:56,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:56,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:56,083 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:56,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:56,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:56,083 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:56,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:56,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:56,083 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,084 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:56,084 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:56,084 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:56,084 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:56,084 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:56,184 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:56,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:56,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:56,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:56,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:56,187 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:56,187 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:56,187 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:56,187 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:56,187 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:56,187 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:56,188 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,188 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:56,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:56,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:56,188 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,188 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:56,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:56,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:56,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:56,189 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:56,189 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:56,189 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:56,189 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:56,189 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:56,189 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:56,189 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:56,189 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:56,189 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:56,189 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:56,189 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:56,189 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:56,189 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:56,189 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,189 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:56,190 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:56,190 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:56,190 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:56,190 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:56,190 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:56,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:56,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:56,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:56,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:56,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:56,191 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:56,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3982816b[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:56,191 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:56,192 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:56,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:56,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:56,192 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:56,192 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:56,192 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,192 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:56,192 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:56,193 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:56,193 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,193 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:56,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:56,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:56,193 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:56,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:56,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:56,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:56,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:56,195 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/6650fa4fdb0c48c8ab06cab41b0957ed 2023-04-05 04:57:56,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:56,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:56,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:56,196 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:56,196 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:56,196 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:56,196 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:56,196 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:56,201 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:56,201 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:56,201 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:56,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:56,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:56,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:56,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:56,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:56,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:56,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:56,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:56,202 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,202 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/36095155b883478cacfe3b688ef2ecd3 2023-04-05 04:57:56,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:56,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:56,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:56,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:56,204 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:56,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:56,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,206 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/6e7d181a6dbf4cd2b2fdad77eb9fb4a9 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/6e7d181a6dbf4cd2b2fdad77eb9fb4a9 2023-04-05 04:57:56,209 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/6e7d181a6dbf4cd2b2fdad77eb9fb4a9, entries=200, sequenceid=300, filesize=32.3 K 2023-04-05 04:57:56,209 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/6650fa4fdb0c48c8ab06cab41b0957ed as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/6650fa4fdb0c48c8ab06cab41b0957ed 2023-04-05 04:57:56,210 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,210 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:56,210 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:56,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:56,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:56,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:56,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:56,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:56,210 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:56,211 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:56,210 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:56,210 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:56,211 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:56,211 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,211 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:56,211 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:56,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:56,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:56,211 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:56,213 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/6650fa4fdb0c48c8ab06cab41b0957ed, entries=150, sequenceid=300, filesize=11.9 K 2023-04-05 04:57:56,218 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/36095155b883478cacfe3b688ef2ecd3 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/36095155b883478cacfe3b688ef2ecd3 2023-04-05 04:57:56,222 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/36095155b883478cacfe3b688ef2ecd3, entries=150, sequenceid=300, filesize=11.9 K 2023-04-05 04:57:56,222 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 5708f0f53456b130f41c37899ae0e2e0 in 868ms, sequenceid=300, compaction requested=true 2023-04-05 04:57:56,222 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5708f0f53456b130f41c37899ae0e2e0: 2023-04-05 04:57:56,222 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:57:56,223 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-05 04:57:56,223 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-05 04:57:56,223 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:57:56,223 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:57:56,224 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 188886 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-05 04:57:56,224 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): 5708f0f53456b130f41c37899ae0e2e0/A is initiating minor compaction (all files) 2023-04-05 04:57:56,225 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5708f0f53456b130f41c37899ae0e2e0/A in TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:56,225 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/120261db92b2459bb056731a4b4d876c, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/82fdae4e85814080b564d772f54f322f, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/d8e570d7a24b4ffd93e51fce18000d40, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/5b8960c11f32419ab53052f0d9badc4b, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/8091f18f87c946fe9058db4c1abec559, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/558e8a5b750e4a22aeb14d75282c4414, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/6e7d181a6dbf4cd2b2fdad77eb9fb4a9] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp, totalSize=184.5 K 2023-04-05 04:57:56,225 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 84826 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-05 04:57:56,225 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): 5708f0f53456b130f41c37899ae0e2e0/B is initiating minor compaction (all files) 2023-04-05 04:57:56,225 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5708f0f53456b130f41c37899ae0e2e0/B in TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:56,225 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/dcdeac5064d344b6a7c01eb2997a4a73, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/73649c85cb3f44de9fc3d8931b201f48, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/e1897cf0d0a6465db040b8a34876a854, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/24d9708f1b0149b98f9e097bb4154932, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/22ebd28e1e4941218bdbf04b5c91ed77, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/59285e545fe44dfcaf8bb19b56deb184, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/6650fa4fdb0c48c8ab06cab41b0957ed] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp, totalSize=82.8 K 2023-04-05 04:57:56,226 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting dcdeac5064d344b6a7c01eb2997a4a73, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1680670668605 2023-04-05 04:57:56,226 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 120261db92b2459bb056731a4b4d876c, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1680670668605 2023-04-05 04:57:56,226 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 73649c85cb3f44de9fc3d8931b201f48, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1680670669945 2023-04-05 04:57:56,227 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 82fdae4e85814080b564d772f54f322f, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1680670669945 2023-04-05 04:57:56,227 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting e1897cf0d0a6465db040b8a34876a854, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1680670670860 2023-04-05 04:57:56,228 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting d8e570d7a24b4ffd93e51fce18000d40, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1680670670860 2023-04-05 04:57:56,228 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 24d9708f1b0149b98f9e097bb4154932, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1680670672056 2023-04-05 04:57:56,228 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 5b8960c11f32419ab53052f0d9badc4b, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1680670672056 2023-04-05 04:57:56,229 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 22ebd28e1e4941218bdbf04b5c91ed77, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1680670672454 2023-04-05 04:57:56,229 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 8091f18f87c946fe9058db4c1abec559, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1680670672454 2023-04-05 04:57:56,229 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 59285e545fe44dfcaf8bb19b56deb184, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1680670673155 2023-04-05 04:57:56,229 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 558e8a5b750e4a22aeb14d75282c4414, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1680670673155 2023-04-05 04:57:56,229 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 6650fa4fdb0c48c8ab06cab41b0957ed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1680670674230 2023-04-05 04:57:56,229 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 6e7d181a6dbf4cd2b2fdad77eb9fb4a9, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1680670674230 2023-04-05 04:57:56,242 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): 5708f0f53456b130f41c37899ae0e2e0#B#compaction#332 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:57:56,249 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5708f0f53456b130f41c37899ae0e2e0#A#compaction#333 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:57:56,311 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:56,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:56,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:56,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:56,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:56,313 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:56,314 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:56,314 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:56,314 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:56,314 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:56,314 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:56,315 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:56,315 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:56,315 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,315 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:56,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:56,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:56,316 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:56,316 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:56,316 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:56,316 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:56,316 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:56,316 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:56,316 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:56,316 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:56,316 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:56,316 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:56,316 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5708f0f53456b130f41c37899ae0e2e0 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-05 04:57:56,316 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=A 2023-04-05 04:57:56,317 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:56,317 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=B 2023-04-05 04:57:56,317 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:56,317 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=C 2023-04-05 04:57:56,317 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:56,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:57:56,561 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:56,581 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:56,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54608 deadline: 1680670736575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:56,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670736577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,589 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:56,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54584 deadline: 1680670736578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,593 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:56,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54542 deadline: 1680670736582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,661 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/95898a397f9640618c637a5601336bb3 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/95898a397f9640618c637a5601336bb3 2023-04-05 04:57:56,665 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 5708f0f53456b130f41c37899ae0e2e0/B of 5708f0f53456b130f41c37899ae0e2e0 into 95898a397f9640618c637a5601336bb3(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:57:56,665 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5708f0f53456b130f41c37899ae0e2e0: 2023-04-05 04:57:56,665 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0., storeName=5708f0f53456b130f41c37899ae0e2e0/B, priority=9, startTime=1680670676223; duration=0sec 2023-04-05 04:57:56,665 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:57:56,665 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-05 04:57:56,669 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 84826 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-05 04:57:56,669 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): 5708f0f53456b130f41c37899ae0e2e0/C is initiating minor compaction (all files) 2023-04-05 04:57:56,669 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5708f0f53456b130f41c37899ae0e2e0/C in TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:56,669 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/aea24ad35b8c45369808515755e0280a, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/749447674c724ffc9b84e40292912534, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/6c55c83d84d14ce194684ce3736bb1cb, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/6fea3ce1ff1c4ad8a334b125f8c98fbb, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/7da440b195724986b95c74e2da4ee3d7, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/fc7441fe6c0c4f4dbfef4b2483217522, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/36095155b883478cacfe3b688ef2ecd3] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp, totalSize=82.8 K 2023-04-05 04:57:56,670 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting aea24ad35b8c45369808515755e0280a, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1680670668605 2023-04-05 04:57:56,670 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 749447674c724ffc9b84e40292912534, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1680670669945 2023-04-05 04:57:56,670 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 6c55c83d84d14ce194684ce3736bb1cb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1680670670860 2023-04-05 04:57:56,671 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 6fea3ce1ff1c4ad8a334b125f8c98fbb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1680670672056 2023-04-05 04:57:56,671 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 7da440b195724986b95c74e2da4ee3d7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1680670672454 2023-04-05 04:57:56,672 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting fc7441fe6c0c4f4dbfef4b2483217522, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1680670673155 2023-04-05 04:57:56,672 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/95eb8580931b40c0ab87bf00698a0a9b as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/95eb8580931b40c0ab87bf00698a0a9b 2023-04-05 04:57:56,672 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 36095155b883478cacfe3b688ef2ecd3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1680670674230 2023-04-05 04:57:56,675 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 5708f0f53456b130f41c37899ae0e2e0/A of 5708f0f53456b130f41c37899ae0e2e0 into 95eb8580931b40c0ab87bf00698a0a9b(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:57:56,675 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5708f0f53456b130f41c37899ae0e2e0: 2023-04-05 04:57:56,675 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0., storeName=5708f0f53456b130f41c37899ae0e2e0/A, priority=9, startTime=1680670676222; duration=0sec 2023-04-05 04:57:56,675 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:57:56,686 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): 5708f0f53456b130f41c37899ae0e2e0#C#compaction#335 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:57:56,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:56,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54608 deadline: 1680670736683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:56,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670736686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,698 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:56,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54584 deadline: 1680670736690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,698 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:56,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54542 deadline: 1680670736694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,701 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/bb252bc8b06249038e5820912d8cf363 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/bb252bc8b06249038e5820912d8cf363 2023-04-05 04:57:56,704 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 5708f0f53456b130f41c37899ae0e2e0/C of 5708f0f53456b130f41c37899ae0e2e0 into bb252bc8b06249038e5820912d8cf363(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:57:56,704 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5708f0f53456b130f41c37899ae0e2e0: 2023-04-05 04:57:56,704 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0., storeName=5708f0f53456b130f41c37899ae0e2e0/C, priority=9, startTime=1680670676223; duration=0sec 2023-04-05 04:57:56,705 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:57:56,731 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304053b2d54aefc4a444d9561397237e6b152 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304053b2d54aefc4a444d9561397237e6b152 2023-04-05 04:57:56,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:56,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54608 deadline: 1680670736889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:56,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670736890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,902 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:56,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54584 deadline: 1680670736900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:56,903 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:56,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54542 deadline: 1680670736900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:57,136 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=319, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/1d436fb6cbc14a15a05faa18f645df94 2023-04-05 04:57:57,197 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:57,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54608 deadline: 1680670737196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:57,198 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:57,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670737196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:57,207 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:57,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54542 deadline: 1680670737204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:57,207 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:57,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54584 deadline: 1680670737204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:57,283 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:57,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54552 deadline: 1680670737282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:57,285 DEBUG [Thread-2228] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4168 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0., hostname=jenkins-hbase20.apache.org,38091,1680670569206, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 04:57:57,545 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=319 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/1d6863f5fa91410e9c67a3e969297d63 2023-04-05 04:57:57,702 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:57,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670737699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:57,702 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:57,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54608 deadline: 1680670737699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:57,717 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:57,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54542 deadline: 1680670737711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:57,717 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:57,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54584 deadline: 1680670737712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:57,954 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=319 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/c996e97d8ac7481c90356b3e2d51401a 2023-04-05 04:57:57,958 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/1d436fb6cbc14a15a05faa18f645df94 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/1d436fb6cbc14a15a05faa18f645df94 2023-04-05 04:57:57,960 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/1d436fb6cbc14a15a05faa18f645df94, entries=150, sequenceid=319, filesize=25.4 K 2023-04-05 04:57:57,961 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/1d6863f5fa91410e9c67a3e969297d63 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/1d6863f5fa91410e9c67a3e969297d63 2023-04-05 04:57:57,964 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/1d6863f5fa91410e9c67a3e969297d63, entries=150, sequenceid=319, filesize=11.9 K 2023-04-05 04:57:57,964 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/c996e97d8ac7481c90356b3e2d51401a as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/c996e97d8ac7481c90356b3e2d51401a 2023-04-05 04:57:57,967 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/c996e97d8ac7481c90356b3e2d51401a, entries=150, sequenceid=319, filesize=11.9 K 2023-04-05 04:57:57,968 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 5708f0f53456b130f41c37899ae0e2e0 in 1652ms, sequenceid=319, compaction requested=false 2023-04-05 04:57:57,968 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5708f0f53456b130f41c37899ae0e2e0: 2023-04-05 04:57:57,968 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:57,968 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:57,968 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:57,968 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:57,969 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:57,969 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:57,969 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:57,969 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:57,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:57,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:57,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:57,970 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:57,970 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:57,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:57,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:57,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:57,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:57,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:57,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:57,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ee083c7[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:57,971 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:57,971 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:57,971 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:57,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:57,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:57,972 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:57,972 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:57,972 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:57,972 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:57,972 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:57,972 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:57,972 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:57,973 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:57,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:57,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:57,973 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:57,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:57,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:57,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:57,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:57,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:57,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:57,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:57,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:57,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:57,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:57,975 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:57,975 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:57,975 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:57,975 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:57,975 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:57,975 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:57,976 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:57,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:57,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:57,976 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:57,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:57,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:57,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:57,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:57,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:57,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:57,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:57,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:57,977 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:57,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:57,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:57,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:57,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:57,978 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:57,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:57,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:57,979 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:57,979 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:57,979 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:57,979 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:57,979 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:57,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:57,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:57,979 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:57,979 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:57,979 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:57,979 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:57,979 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:57,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:57,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:57,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:57,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:57,980 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:57,980 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:58,081 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:58,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:58,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:58,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:58,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:58,083 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:58,083 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:58,083 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:58,083 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:58,084 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:58,084 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:58,084 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:58,084 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:58,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:58,084 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:58,084 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:58,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:58,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:58,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:58,085 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:58,086 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:58,086 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:58,086 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:58,086 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:58,086 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:58,086 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:58,086 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:58,086 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:58,086 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:58,086 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5708f0f53456b130f41c37899ae0e2e0 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-05 04:57:58,086 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=A 2023-04-05 04:57:58,086 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:58,086 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=B 2023-04-05 04:57:58,086 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:58,086 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=C 2023-04-05 04:57:58,086 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:58,503 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040521101bc38d344f7d848da1ceb65f1229 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040521101bc38d344f7d848da1ceb65f1229 2023-04-05 04:57:58,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:57:58,711 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:58,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:58,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670738743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:58,752 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:58,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54584 deadline: 1680670738745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:58,752 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:58,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54542 deadline: 1680670738746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:58,752 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:58,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54608 deadline: 1680670738750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:58,854 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:58,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670738852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:58,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:58,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54584 deadline: 1680670738852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:58,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:58,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54542 deadline: 1680670738852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:58,856 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:58,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54608 deadline: 1680670738853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:58,907 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=335, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/3f72997246bc48d39cbd20781a4ad868 2023-04-05 04:57:59,061 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:59,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54584 deadline: 1680670739057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,062 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:59,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670739057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,062 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:59,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54542 deadline: 1680670739057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,062 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:59,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54608 deadline: 1680670739057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,323 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/d9fb0cf85f884d57980a7092bcbc512f 2023-04-05 04:57:59,336 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/609b773b588c4c798ee20b3b17ca8755 2023-04-05 04:57:59,341 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/3f72997246bc48d39cbd20781a4ad868 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/3f72997246bc48d39cbd20781a4ad868 2023-04-05 04:57:59,349 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/3f72997246bc48d39cbd20781a4ad868, entries=150, sequenceid=335, filesize=25.4 K 2023-04-05 04:57:59,350 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/d9fb0cf85f884d57980a7092bcbc512f as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/d9fb0cf85f884d57980a7092bcbc512f 2023-04-05 04:57:59,356 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/d9fb0cf85f884d57980a7092bcbc512f, entries=150, sequenceid=335, filesize=11.9 K 2023-04-05 04:57:59,356 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/609b773b588c4c798ee20b3b17ca8755 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/609b773b588c4c798ee20b3b17ca8755 2023-04-05 04:57:59,361 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/609b773b588c4c798ee20b3b17ca8755, entries=150, sequenceid=335, filesize=11.9 K 2023-04-05 04:57:59,361 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 5708f0f53456b130f41c37899ae0e2e0 in 1275ms, sequenceid=335, compaction requested=true 2023-04-05 04:57:59,361 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5708f0f53456b130f41c37899ae0e2e0: 2023-04-05 04:57:59,361 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:59,361 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:59,361 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:59,361 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,362 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:59,362 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:59,363 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,363 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,363 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:59,365 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:59,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:59,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:59,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:59,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:59,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:57:59,367 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5708f0f53456b130f41c37899ae0e2e0 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-05 04:57:59,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,367 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=A 2023-04-05 04:57:59,367 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:59,367 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=B 2023-04-05 04:57:59,367 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:59,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:59,367 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=C 2023-04-05 04:57:59,367 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:57:59,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:59,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6eddeedf[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:59,368 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:59,368 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,376 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,377 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,377 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:59,377 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:59,377 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:59,377 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:59,380 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,380 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:59,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:59,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:59,380 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:59,380 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:59,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:59,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:59,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:59,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:59,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:59,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,382 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304052d813e4338f44ff5b0da21493d95dae6 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304052d813e4338f44ff5b0da21493d95dae6 2023-04-05 04:57:59,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:59,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:59,383 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:59,383 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:59,383 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:59,383 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:59,383 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:59,384 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:59,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54608 deadline: 1680670739377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,384 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:59,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54542 deadline: 1680670739379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,394 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:59,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670739381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,394 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:59,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54584 deadline: 1680670739382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,395 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:59,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:59,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:59,396 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:59,396 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,396 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:59,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:59,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,397 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:59,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:59,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,399 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:59,399 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:59,399 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:59,399 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:59,399 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:59,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:59,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:59,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:59,399 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:59,400 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,400 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,400 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,400 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:59,400 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:59,400 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:59,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:59,497 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:59,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54542 deadline: 1680670739495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,497 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:59,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54608 deadline: 1680670739495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,497 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:59,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670739496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,500 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:59,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:59,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:59,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:59,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54584 deadline: 1680670739497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:59,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:59,503 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:59,503 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:59,503 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:59,503 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:59,503 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,503 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:59,504 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,504 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:59,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:59,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:59,505 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,505 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:59,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:59,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:59,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:59,506 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:59,506 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:59,506 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:59,506 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:59,506 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:59,506 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:59,506 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:59,506 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:59,506 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:59,506 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:59,506 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:59,506 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:59,506 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:59,507 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:59,507 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,507 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:59,507 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:59,507 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:59,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:59,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:59,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:59,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:59,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,508 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:59,509 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:59,509 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:59,509 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:59,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1468cf2b[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:59,509 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,509 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,510 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:59,510 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,510 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:59,510 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:59,510 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:59,510 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:59,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:59,511 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:59,511 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:59,511 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:59,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:59,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:59,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:59,513 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:59,513 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:59,514 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:59,514 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:59,514 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:59,514 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,514 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:59,515 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:59,515 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:59,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:59,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:59,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:59,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:59,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,516 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:59,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:59,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,518 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:59,518 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:59,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:59,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:59,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:59,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:59,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:59,518 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:59,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:59,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:59,518 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:59,518 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:59,519 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,519 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:59,519 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:59,519 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,519 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,519 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,519 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,618 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:59,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:59,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:59,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:59,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:59,621 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:59,621 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:59,621 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:59,621 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:59,622 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,622 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:59,624 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,624 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:59,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:59,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:59,625 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,625 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:59,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:59,625 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:59,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:59,625 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:59,625 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:59,626 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:59,626 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:59,626 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:59,626 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:59,626 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:59,626 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:59,626 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:59,626 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:59,626 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:59,626 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:59,626 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:59,626 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:59,626 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,626 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:59,626 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:59,627 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,627 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:59,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:59,627 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,628 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:59,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:59,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:59,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:59,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:59,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:59,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39cc8d14[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:59,631 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:59,631 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,632 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,632 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:59,632 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:59,632 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:59,632 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,632 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:59,633 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,633 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:59,633 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:59,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:59,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:59,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:59,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:59,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:59,633 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:59,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:59,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:59,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:59,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:59,643 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:59,643 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:59,643 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:59,643 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:59,643 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:59,645 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,645 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:59,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:59,645 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:59,645 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:59,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:59,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:59,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:59,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:59,646 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:59,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:59,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:59,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,648 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:59,648 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:59,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:59,648 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:59,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:59,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:59,648 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,648 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:59,648 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:59,649 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:59,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:59,649 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:59,649 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:59,649 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,649 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,649 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:59,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:59,706 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:59,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54608 deadline: 1680670739698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:59,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54542 deadline: 1680670739698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:59,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670739698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:57:59,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54584 deadline: 1680670739703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,749 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:59,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:59,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:59,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:59,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:59,752 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:59,752 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:59,752 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:59,752 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:59,752 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,752 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:59,753 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,753 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:59,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:59,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:59,753 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,753 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:59,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:59,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:59,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:59,754 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:59,758 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:59,758 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:59,758 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:59,758 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:59,758 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:59,763 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:59,763 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:59,763 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:59,764 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:59,764 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:59,764 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:59,764 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:59,764 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:59,764 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,764 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:59,764 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:59,765 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,765 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:59,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:59,765 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,765 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:59,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:59,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:59,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:59,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:59,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:59,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63e8df51[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:59,768 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:59,768 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,769 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,769 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:59,769 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:59,769 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:59,769 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,769 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:59,770 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:59,770 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:59,770 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,770 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:59,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:59,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:59,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:59,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:59,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:59,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:59,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:59,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:59,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:59,779 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:59,779 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:59,779 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:59,779 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:59,779 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:59,785 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,785 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,785 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:59,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:59,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:59,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:59,785 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:59,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:59,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:59,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:59,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:59,787 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:59,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:59,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,789 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:59,789 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:59,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:59,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:59,789 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,789 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:59,789 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:59,789 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,789 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,789 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,789 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,789 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:59,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:59,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:59,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:59,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:59,790 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:59,790 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:59,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:59,801 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=355, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/3fd1c30ffbe641b6b9e5d15e6872550a 2023-04-05 04:57:59,890 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:57:59,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:57:59,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:57:59,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:57:59,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:57:59,893 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:57:59,893 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:59,893 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:57:59,893 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:57:59,894 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,894 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:59,894 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,894 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:59,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:59,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:59,895 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,895 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:57:59,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:59,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:57:59,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:59,895 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:57:59,898 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:57:59,898 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:57:59,898 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:57:59,898 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:57:59,898 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:57:59,898 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:57:59,898 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:57:59,898 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:59,898 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:57:59,898 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:57:59,898 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:57:59,898 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:57:59,898 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:57:59,898 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,898 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:57:59,899 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:57:59,900 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,900 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,900 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,900 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:57:59,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:59,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:59,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:59,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:59,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:59,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:59,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:57:59,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62426ee5[Count = 0] remaining members to acquire global barrier 2023-04-05 04:57:59,901 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:57:59,901 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,902 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,902 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,902 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:57:59,902 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:57:59,902 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:57:59,902 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:57:59,903 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:59,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:59,903 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:57:59,903 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:59,903 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:57:59,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:59,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:59,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:59,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:59,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:59,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:57:59,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:57:59,905 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:57:59,905 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:57:59,905 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:57:59,905 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:57:59,905 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:57:59,907 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,907 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:59,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:59,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:57:59,908 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,908 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:57:59,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:57:59,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:57:59,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:59,909 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:57:59,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:57:59,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:57:59,910 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,910 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:59,910 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:57:59,910 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:57:59,910 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:57:59,910 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,910 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:57:59,910 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:57:59,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:57:59,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:57:59,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:57:59,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:57:59,910 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:57:59,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:57:59,911 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:57:59,911 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:57:59,911 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:57:59,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:57:59,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:00,011 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:00,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:00,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:00,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:00,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:00,014 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:00,015 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:00,015 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:00,015 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:00,015 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:00,015 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:00,015 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,015 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:00,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:00,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:00,016 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,016 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:00,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:00,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:00,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:00,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:00,016 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:00,017 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:00,017 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:00,017 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:00,017 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:00,017 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:00,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54608 deadline: 1680670740009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,017 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:00,017 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:00,017 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:00,017 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:58:00,017 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:00,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54542 deadline: 1680670740010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,017 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:58:00,017 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:00,017 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:58:00,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670740010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,017 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:58:00,017 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:00,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:00,018 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:00,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54584 deadline: 1680670740014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,018 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,018 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:00,018 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:00,019 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,019 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:00,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:00,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:00,019 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:00,020 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:00,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:00,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:00,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:00,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:00,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:00,021 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:00,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3081e411[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:00,021 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:00,021 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:00,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:00,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:00,022 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:00,022 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:00,022 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:00,022 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,022 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:00,022 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:00,022 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:00,023 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:00,023 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:00,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:00,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:00,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:00,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:00,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:00,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:00,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:00,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:00,031 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:00,031 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:00,031 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:00,031 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:00,031 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:00,035 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:00,035 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:00,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:00,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:00,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:00,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:00,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:00,038 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:00,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:00,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:00,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:00,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:00,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:00,040 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:00,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,041 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:00,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:00,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,042 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:00,042 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:00,042 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:00,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:00,042 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:00,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:00,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:00,042 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:00,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:00,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:00,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:00,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:00,043 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:00,043 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:00,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:00,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:00,143 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:00,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:00,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:00,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:00,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:00,146 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:00,146 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:00,146 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:00,146 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:00,146 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:00,146 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:00,149 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:00,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:00,149 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:00,150 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:00,150 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:00,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:00,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:00,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:00,151 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:00,151 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:00,151 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:00,151 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:00,151 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:00,151 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:00,152 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:00,152 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:00,152 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:58:00,153 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:58:00,153 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:58:00,153 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:58:00,153 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:00,153 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:00,153 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,153 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:00,153 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:00,154 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,154 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:00,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:00,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:00,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:00,154 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:00,154 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:00,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:00,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:00,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:00,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:00,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6df87b38[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:00,155 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:00,155 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:00,156 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:00,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:00,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:00,156 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:00,156 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:00,156 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:00,157 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:00,157 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:00,157 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:00,159 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:00,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:00,159 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,159 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:00,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:00,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:00,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:00,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:00,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:00,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:00,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:00,161 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:00,161 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:00,161 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:00,161 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:00,161 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:00,162 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:00,162 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:00,162 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:00,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:00,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:00,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:00,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:00,163 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:00,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:00,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:00,164 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:00,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,165 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,165 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:00,165 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:00,165 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,165 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:00,165 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:00,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:00,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:00,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:00,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:00,165 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:00,165 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:00,166 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:00,166 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:00,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:00,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:00,166 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:00,166 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:00,217 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=355 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/48448622064e4b269545dabcc40fa32f 2023-04-05 04:58:00,230 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=355 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/36d4882e95c44a55abe238831c94e74c 2023-04-05 04:58:00,235 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/3fd1c30ffbe641b6b9e5d15e6872550a as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/3fd1c30ffbe641b6b9e5d15e6872550a 2023-04-05 04:58:00,240 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/3fd1c30ffbe641b6b9e5d15e6872550a, entries=200, sequenceid=355, filesize=32.3 K 2023-04-05 04:58:00,241 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/48448622064e4b269545dabcc40fa32f as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/48448622064e4b269545dabcc40fa32f 2023-04-05 04:58:00,245 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/48448622064e4b269545dabcc40fa32f, entries=150, sequenceid=355, filesize=11.9 K 2023-04-05 04:58:00,246 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/36d4882e95c44a55abe238831c94e74c as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/36d4882e95c44a55abe238831c94e74c 2023-04-05 04:58:00,249 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/36d4882e95c44a55abe238831c94e74c, entries=150, sequenceid=355, filesize=11.9 K 2023-04-05 04:58:00,250 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 5708f0f53456b130f41c37899ae0e2e0 in 883ms, sequenceid=355, compaction requested=true 2023-04-05 04:58:00,250 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5708f0f53456b130f41c37899ae0e2e0: 2023-04-05 04:58:00,250 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:58:00,250 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:58:00,250 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:58:00,250 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-05 04:58:00,250 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:58:00,252 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111736 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:58:00,252 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): 5708f0f53456b130f41c37899ae0e2e0/A is initiating minor compaction (all files) 2023-04-05 04:58:00,252 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5708f0f53456b130f41c37899ae0e2e0/A in TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:58:00,252 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/95eb8580931b40c0ab87bf00698a0a9b, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/1d436fb6cbc14a15a05faa18f645df94, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/3f72997246bc48d39cbd20781a4ad868, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/3fd1c30ffbe641b6b9e5d15e6872550a] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp, totalSize=109.1 K 2023-04-05 04:58:00,252 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49256 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:58:00,253 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): 5708f0f53456b130f41c37899ae0e2e0/B is initiating minor compaction (all files) 2023-04-05 04:58:00,253 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5708f0f53456b130f41c37899ae0e2e0/B in TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:58:00,253 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/95898a397f9640618c637a5601336bb3, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/1d6863f5fa91410e9c67a3e969297d63, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/d9fb0cf85f884d57980a7092bcbc512f, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/48448622064e4b269545dabcc40fa32f] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp, totalSize=48.1 K 2023-04-05 04:58:00,253 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 95eb8580931b40c0ab87bf00698a0a9b, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1680670674230 2023-04-05 04:58:00,253 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 95898a397f9640618c637a5601336bb3, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1680670674230 2023-04-05 04:58:00,253 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 1d436fb6cbc14a15a05faa18f645df94, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1680670675423 2023-04-05 04:58:00,253 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 1d6863f5fa91410e9c67a3e969297d63, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1680670675423 2023-04-05 04:58:00,253 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 3f72997246bc48d39cbd20781a4ad868, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1680670676573 2023-04-05 04:58:00,254 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting d9fb0cf85f884d57980a7092bcbc512f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1680670676573 2023-04-05 04:58:00,254 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 3fd1c30ffbe641b6b9e5d15e6872550a, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1680670678744 2023-04-05 04:58:00,254 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 48448622064e4b269545dabcc40fa32f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1680670678745 2023-04-05 04:58:00,266 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:00,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:00,269 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): 5708f0f53456b130f41c37899ae0e2e0#B#compaction#345 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:58:00,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:00,284 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5708f0f53456b130f41c37899ae0e2e0#A#compaction#344 average throughput is 0.90 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:58:00,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:00,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:00,310 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:00,310 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:00,310 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:00,310 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:00,312 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:00,312 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:00,312 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,312 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:00,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:00,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:00,313 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,313 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:00,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:00,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:00,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:00,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:00,314 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:00,350 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:00,350 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:00,350 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:00,350 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:00,350 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:00,358 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:00,358 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:00,358 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:58:00,360 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:58:00,361 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5708f0f53456b130f41c37899ae0e2e0 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-05 04:58:00,361 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=A 2023-04-05 04:58:00,361 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:00,361 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=B 2023-04-05 04:58:00,361 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:00,361 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=C 2023-04-05 04:58:00,361 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:00,417 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304058bf46da8b1aa43138b307792c41010f8 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304058bf46da8b1aa43138b307792c41010f8 2023-04-05 04:58:00,446 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=367, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/b7e9e9fdd5a9415f850692f73c2d5e15 2023-04-05 04:58:00,523 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:58:00,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:58:00,569 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:00,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54584 deadline: 1680670740560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:00,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54608 deadline: 1680670740568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,580 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:00,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54542 deadline: 1680670740569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,581 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:00,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670740569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,676 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:00,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54584 deadline: 1680670740671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,683 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:00,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54608 deadline: 1680670740676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,690 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:00,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54542 deadline: 1680670740681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,690 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:00,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670740682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,739 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/dd443fd6f0554ccaa25aa8f305ec4e58 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/dd443fd6f0554ccaa25aa8f305ec4e58 2023-04-05 04:58:00,743 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5708f0f53456b130f41c37899ae0e2e0/B of 5708f0f53456b130f41c37899ae0e2e0 into dd443fd6f0554ccaa25aa8f305ec4e58(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:58:00,743 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5708f0f53456b130f41c37899ae0e2e0: 2023-04-05 04:58:00,743 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0., storeName=5708f0f53456b130f41c37899ae0e2e0/B, priority=12, startTime=1680670680250; duration=0sec 2023-04-05 04:58:00,744 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:58:00,744 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:58:00,747 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49256 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:58:00,747 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): 5708f0f53456b130f41c37899ae0e2e0/C is initiating minor compaction (all files) 2023-04-05 04:58:00,747 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5708f0f53456b130f41c37899ae0e2e0/C in TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:58:00,748 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/bb252bc8b06249038e5820912d8cf363, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/c996e97d8ac7481c90356b3e2d51401a, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/609b773b588c4c798ee20b3b17ca8755, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/36d4882e95c44a55abe238831c94e74c] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp, totalSize=48.1 K 2023-04-05 04:58:00,748 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting bb252bc8b06249038e5820912d8cf363, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1680670674230 2023-04-05 04:58:00,748 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting c996e97d8ac7481c90356b3e2d51401a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1680670675423 2023-04-05 04:58:00,749 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 609b773b588c4c798ee20b3b17ca8755, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1680670676573 2023-04-05 04:58:00,749 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 36d4882e95c44a55abe238831c94e74c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1680670678745 2023-04-05 04:58:00,772 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): 5708f0f53456b130f41c37899ae0e2e0#C#compaction#348 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:58:00,800 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/012d92c768804da18254698936817ae2 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/012d92c768804da18254698936817ae2 2023-04-05 04:58:00,804 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5708f0f53456b130f41c37899ae0e2e0/A of 5708f0f53456b130f41c37899ae0e2e0 into 012d92c768804da18254698936817ae2(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:58:00,804 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5708f0f53456b130f41c37899ae0e2e0: 2023-04-05 04:58:00,804 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0., storeName=5708f0f53456b130f41c37899ae0e2e0/A, priority=12, startTime=1680670680250; duration=0sec 2023-04-05 04:58:00,804 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:58:00,871 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/bbd262734bcd454696df8e2e94177135 2023-04-05 04:58:00,882 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:00,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54584 deadline: 1680670740877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,892 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:00,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54608 deadline: 1680670740885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,892 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:00,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54542 deadline: 1680670740891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,893 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:00,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54616 deadline: 1680670740891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,904 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/767d832afe8f4829914a414b397bd9f2 2023-04-05 04:58:00,912 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/b7e9e9fdd5a9415f850692f73c2d5e15 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/b7e9e9fdd5a9415f850692f73c2d5e15 2023-04-05 04:58:00,916 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/b7e9e9fdd5a9415f850692f73c2d5e15, entries=150, sequenceid=367, filesize=25.4 K 2023-04-05 04:58:00,917 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/bbd262734bcd454696df8e2e94177135 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/bbd262734bcd454696df8e2e94177135 2023-04-05 04:58:00,923 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/bbd262734bcd454696df8e2e94177135, entries=150, sequenceid=367, filesize=11.9 K 2023-04-05 04:58:00,924 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/767d832afe8f4829914a414b397bd9f2 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/767d832afe8f4829914a414b397bd9f2 2023-04-05 04:58:00,932 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/767d832afe8f4829914a414b397bd9f2, entries=150, sequenceid=367, filesize=11.9 K 2023-04-05 04:58:00,933 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 5708f0f53456b130f41c37899ae0e2e0 in 572ms, sequenceid=367, compaction requested=false 2023-04-05 04:58:00,933 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5708f0f53456b130f41c37899ae0e2e0: 2023-04-05 04:58:00,933 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:58:00,933 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:00,933 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:00,933 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,934 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:00,934 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:00,935 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,935 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:00,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:00,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:00,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:00,935 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:00,935 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:00,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:00,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:00,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:00,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:00,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48cf3381[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:00,936 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:00,936 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:00,937 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:00,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:00,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:00,937 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:00,937 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:00,937 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:00,937 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,937 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:00,938 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,938 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:00,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:00,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:00,938 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:00,938 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:00,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:00,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:00,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:00,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:00,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:00,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:00,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:00,941 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:00,941 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:00,941 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:00,942 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:00,942 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:00,942 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:00,942 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:00,942 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:00,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:00,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:00,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:00,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:00,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:00,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:00,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:00,943 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:00,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:00,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:00,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:00,944 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:00,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:00,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:00,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,960 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:00,960 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:00,960 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:00,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:00,960 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:00,960 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:00,960 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:00,960 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:00,960 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:00,960 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:00,960 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:00,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:00,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:00,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:00,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:00,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:00,960 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:00,961 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:01,061 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:01,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:01,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:01,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:01,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:01,066 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:01,066 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:01,067 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:01,067 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:01,067 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:01,067 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:01,068 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:01,068 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:01,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:01,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:01,068 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:01,068 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:01,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:01,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:01,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:01,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:01,069 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:01,069 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:01,069 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:01,069 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:01,069 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:01,069 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:01,069 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:01,069 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:01,069 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:58:01,069 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. started... 2023-04-05 04:58:01,069 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5708f0f53456b130f41c37899ae0e2e0 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-05 04:58:01,070 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=A 2023-04-05 04:58:01,070 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:01,070 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=B 2023-04-05 04:58:01,070 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:01,070 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=C 2023-04-05 04:58:01,070 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:01,113 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304056a76542a5b1e436b9a11fae2c090ffe2 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304056a76542a5b1e436b9a11fae2c090ffe2 2023-04-05 04:58:01,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:58:01,188 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. as already flushing 2023-04-05 04:58:01,190 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/2a38924bed9a418890d451d18e512af4 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/2a38924bed9a418890d451d18e512af4 2023-04-05 04:58:01,194 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5708f0f53456b130f41c37899ae0e2e0/C of 5708f0f53456b130f41c37899ae0e2e0 into 2a38924bed9a418890d451d18e512af4(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:58:01,194 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5708f0f53456b130f41c37899ae0e2e0: 2023-04-05 04:58:01,194 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0., storeName=5708f0f53456b130f41c37899ae0e2e0/C, priority=12, startTime=1680670680250; duration=0sec 2023-04-05 04:58:01,194 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:58:01,207 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:01,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54584 deadline: 1680670741201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:01,207 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:01,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54608 deadline: 1680670741202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:01,209 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:01,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54616 deadline: 1680670741204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:01,210 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:01,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54542 deadline: 1680670741205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:01,314 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:01,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54608 deadline: 1680670741308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:01,314 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:01,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54584 deadline: 1680670741308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:01,318 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:01,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54616 deadline: 1680670741311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:01,319 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:01,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54542 deadline: 1680670741311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:01,321 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:01,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54552 deadline: 1680670741316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:01,324 DEBUG [Thread-2228] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8207 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0., hostname=jenkins-hbase20.apache.org,38091,1680670569206, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 04:58:01,510 DEBUG [Thread-2247] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x143a1290 to 127.0.0.1:56229 2023-04-05 04:58:01,511 DEBUG [Thread-2247] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:58:01,513 DEBUG [Thread-2245] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x228ff1a4 to 127.0.0.1:56229 2023-04-05 04:58:01,513 DEBUG [Thread-2245] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:58:01,513 DEBUG [Thread-2241] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2ded66ad to 127.0.0.1:56229 2023-04-05 04:58:01,513 DEBUG [Thread-2241] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:58:01,516 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:01,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54584 deadline: 1680670741515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:01,516 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:01,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54608 deadline: 1680670741516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:01,520 DEBUG [Thread-2243] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x14d061d2 to 127.0.0.1:56229 2023-04-05 04:58:01,520 DEBUG [Thread-2243] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:58:01,521 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:01,521 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:01,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54616 deadline: 1680670741521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:01,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54542 deadline: 1680670741521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:01,523 DEBUG [Thread-2239] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x36a9ece3 to 127.0.0.1:56229 2023-04-05 04:58:01,523 DEBUG [Thread-2239] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:58:01,538 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=389, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/e6a675f6432f4952976fe3a4a007e084 2023-04-05 04:58:01,819 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:01,820 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:01,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54584 deadline: 1680670741819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:01,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54608 deadline: 1680670741819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:01,822 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:01,822 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:01,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54616 deadline: 1680670741822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:01,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54542 deadline: 1680670741822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:01,949 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=389 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/caadd1da5943447dbb87e2b04bac9354 2023-04-05 04:58:02,322 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:02,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54608 deadline: 1680670742322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:02,325 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:02,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:02,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54542 deadline: 1680670742325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:02,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54584 deadline: 1680670742325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:02,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:02,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54616 deadline: 1680670742327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:02,359 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=389 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/3f960165605843679b6d8d5da520b8ba 2023-04-05 04:58:02,363 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/e6a675f6432f4952976fe3a4a007e084 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/e6a675f6432f4952976fe3a4a007e084 2023-04-05 04:58:02,366 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/e6a675f6432f4952976fe3a4a007e084, entries=150, sequenceid=389, filesize=25.4 K 2023-04-05 04:58:02,367 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/caadd1da5943447dbb87e2b04bac9354 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/caadd1da5943447dbb87e2b04bac9354 2023-04-05 04:58:02,370 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/caadd1da5943447dbb87e2b04bac9354, entries=150, sequenceid=389, filesize=11.9 K 2023-04-05 04:58:02,371 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/3f960165605843679b6d8d5da520b8ba as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/3f960165605843679b6d8d5da520b8ba 2023-04-05 04:58:02,374 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/3f960165605843679b6d8d5da520b8ba, entries=150, sequenceid=389, filesize=11.9 K 2023-04-05 04:58:02,375 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 5708f0f53456b130f41c37899ae0e2e0 in 1306ms, sequenceid=389, compaction requested=true 2023-04-05 04:58:02,375 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5708f0f53456b130f41c37899ae0e2e0: 2023-04-05 04:58:02,375 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:58:02,375 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:02,375 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:02,375 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:02,375 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:02,375 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:02,378 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:02,378 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:02,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:02,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:02,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:02,378 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:02,378 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:02,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:02,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:02,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:02,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:02,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:02,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:02,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@398b493b[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:02,379 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:02,379 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:02,379 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:02,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:02,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:02,379 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:02,380 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:02,380 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:02,380 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:02,380 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:02,380 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:02,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:02,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:02,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:02,380 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:02,380 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:02,380 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:02,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:02,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:02,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:02,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:02,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:02,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:02,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:02,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:02,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:02,386 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:02,386 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:02,386 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:02,386 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:02,387 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:02,393 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:02,393 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:02,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:02,393 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:02,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:02,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:02,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:02,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:02,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:02,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:02,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:02,394 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:02,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:02,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:02,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:02,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:02,395 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:02,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:02,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:02,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:02,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:02,397 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:02,397 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:02,397 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:02,397 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:02,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:02,397 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:02,397 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:02,397 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:02,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:02,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:02,397 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:02,397 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:02,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:02,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:02,397 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:02,397 DEBUG [Thread-2238] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:02,397 DEBUG [Thread-2238] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:02,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:02,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:02,497 DEBUG [Thread-2238] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:02,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:03,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:58:03,325 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5708f0f53456b130f41c37899ae0e2e0 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-05 04:58:03,326 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=A 2023-04-05 04:58:03,326 DEBUG [Thread-2236] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x060c5cba to 127.0.0.1:56229 2023-04-05 04:58:03,326 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:03,326 DEBUG [Thread-2236] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:58:03,326 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=B 2023-04-05 04:58:03,326 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:03,326 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=C 2023-04-05 04:58:03,326 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:03,331 DEBUG [Thread-2232] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5da5030e to 127.0.0.1:56229 2023-04-05 04:58:03,331 DEBUG [Thread-2232] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:58:03,334 DEBUG [Thread-2230] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x57b6628e to 127.0.0.1:56229 2023-04-05 04:58:03,335 DEBUG [Thread-2230] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:58:03,336 DEBUG [Thread-2234] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x74d6bf18 to 127.0.0.1:56229 2023-04-05 04:58:03,336 DEBUG [Thread-2234] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:58:03,745 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304055788813340c24ab9b668dbde9020be2e to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304055788813340c24ab9b668dbde9020be2e 2023-04-05 04:58:04,152 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=403, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/a5db877b79664c97a148a4251ce113bf 2023-04-05 04:58:04,561 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=403 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/f01daa9c541f43ba97535e4bd52f72c6 2023-04-05 04:58:04,971 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=403 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/52c7d1513a39427fa09c2f8e02f409e0 2023-04-05 04:58:04,975 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/a5db877b79664c97a148a4251ce113bf as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/a5db877b79664c97a148a4251ce113bf 2023-04-05 04:58:04,978 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/a5db877b79664c97a148a4251ce113bf, entries=150, sequenceid=403, filesize=25.4 K 2023-04-05 04:58:04,978 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/f01daa9c541f43ba97535e4bd52f72c6 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/f01daa9c541f43ba97535e4bd52f72c6 2023-04-05 04:58:04,982 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/f01daa9c541f43ba97535e4bd52f72c6, entries=150, sequenceid=403, filesize=11.9 K 2023-04-05 04:58:04,982 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/52c7d1513a39427fa09c2f8e02f409e0 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/52c7d1513a39427fa09c2f8e02f409e0 2023-04-05 04:58:04,986 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/52c7d1513a39427fa09c2f8e02f409e0, entries=150, sequenceid=403, filesize=11.9 K 2023-04-05 04:58:04,987 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=20.13 KB/20610 for 5708f0f53456b130f41c37899ae0e2e0 in 1662ms, sequenceid=403, compaction requested=true 2023-04-05 04:58:04,987 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5708f0f53456b130f41c37899ae0e2e0: 2023-04-05 04:58:04,987 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:58:04,987 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:58:04,987 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:58:04,987 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:58:04,987 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:58:04,988 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 104832 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:58:04,988 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): 5708f0f53456b130f41c37899ae0e2e0/A is initiating minor compaction (all files) 2023-04-05 04:58:04,988 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49392 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:58:04,988 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5708f0f53456b130f41c37899ae0e2e0/A in TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:58:04,988 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): 5708f0f53456b130f41c37899ae0e2e0/B is initiating minor compaction (all files) 2023-04-05 04:58:04,988 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/012d92c768804da18254698936817ae2, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/b7e9e9fdd5a9415f850692f73c2d5e15, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/e6a675f6432f4952976fe3a4a007e084, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/a5db877b79664c97a148a4251ce113bf] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp, totalSize=102.4 K 2023-04-05 04:58:04,988 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5708f0f53456b130f41c37899ae0e2e0/B in TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:58:04,988 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/dd443fd6f0554ccaa25aa8f305ec4e58, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/bbd262734bcd454696df8e2e94177135, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/caadd1da5943447dbb87e2b04bac9354, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/f01daa9c541f43ba97535e4bd52f72c6] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp, totalSize=48.2 K 2023-04-05 04:58:04,989 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 012d92c768804da18254698936817ae2, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1680670678745 2023-04-05 04:58:04,989 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting dd443fd6f0554ccaa25aa8f305ec4e58, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1680670678745 2023-04-05 04:58:04,989 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting b7e9e9fdd5a9415f850692f73c2d5e15, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1680670679376 2023-04-05 04:58:04,989 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting bbd262734bcd454696df8e2e94177135, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1680670679376 2023-04-05 04:58:04,989 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting e6a675f6432f4952976fe3a4a007e084, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=389, earliestPutTs=1680670680554 2023-04-05 04:58:04,989 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting caadd1da5943447dbb87e2b04bac9354, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=389, earliestPutTs=1680670680554 2023-04-05 04:58:04,990 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting a5db877b79664c97a148a4251ce113bf, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1680670681201 2023-04-05 04:58:04,990 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting f01daa9c541f43ba97535e4bd52f72c6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1680670681201 2023-04-05 04:58:05,000 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): 5708f0f53456b130f41c37899ae0e2e0#B#compaction#357 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:58:05,008 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5708f0f53456b130f41c37899ae0e2e0#A#compaction#356 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:58:05,411 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/f9755e36b0604e389803d975bd8d31fb as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/f9755e36b0604e389803d975bd8d31fb 2023-04-05 04:58:05,415 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5708f0f53456b130f41c37899ae0e2e0/B of 5708f0f53456b130f41c37899ae0e2e0 into f9755e36b0604e389803d975bd8d31fb(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:58:05,415 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5708f0f53456b130f41c37899ae0e2e0: 2023-04-05 04:58:05,415 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0., storeName=5708f0f53456b130f41c37899ae0e2e0/B, priority=12, startTime=1680670684987; duration=0sec 2023-04-05 04:58:05,415 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:58:05,416 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:58:05,417 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49392 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:58:05,417 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): 5708f0f53456b130f41c37899ae0e2e0/C is initiating minor compaction (all files) 2023-04-05 04:58:05,417 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5708f0f53456b130f41c37899ae0e2e0/C in TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:58:05,417 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/2a38924bed9a418890d451d18e512af4, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/767d832afe8f4829914a414b397bd9f2, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/3f960165605843679b6d8d5da520b8ba, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/52c7d1513a39427fa09c2f8e02f409e0] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp, totalSize=48.2 K 2023-04-05 04:58:05,417 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 2a38924bed9a418890d451d18e512af4, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1680670678745 2023-04-05 04:58:05,418 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 767d832afe8f4829914a414b397bd9f2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1680670679376 2023-04-05 04:58:05,418 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 3f960165605843679b6d8d5da520b8ba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=389, earliestPutTs=1680670680554 2023-04-05 04:58:05,418 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 52c7d1513a39427fa09c2f8e02f409e0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1680670681201 2023-04-05 04:58:05,426 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): 5708f0f53456b130f41c37899ae0e2e0#C#compaction#358 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:58:05,834 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/d8f313b841404ed7aaec828dbd8b4be4 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/d8f313b841404ed7aaec828dbd8b4be4 2023-04-05 04:58:05,838 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5708f0f53456b130f41c37899ae0e2e0/C of 5708f0f53456b130f41c37899ae0e2e0 into d8f313b841404ed7aaec828dbd8b4be4(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:58:05,838 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5708f0f53456b130f41c37899ae0e2e0: 2023-04-05 04:58:05,838 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0., storeName=5708f0f53456b130f41c37899ae0e2e0/C, priority=12, startTime=1680670684987; duration=0sec 2023-04-05 04:58:05,838 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:58:06,222 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/df9f10d497734953917db7749aec0135 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/df9f10d497734953917db7749aec0135 2023-04-05 04:58:06,229 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5708f0f53456b130f41c37899ae0e2e0/A of 5708f0f53456b130f41c37899ae0e2e0 into df9f10d497734953917db7749aec0135(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-05 04:58:06,230 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5708f0f53456b130f41c37899ae0e2e0: 2023-04-05 04:58:06,230 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0., storeName=5708f0f53456b130f41c37899ae0e2e0/A, priority=12, startTime=1680670684987; duration=1sec 2023-04-05 04:58:06,230 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:58:10,912 DEBUG [regionserver/jenkins-hbase20:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-05 04:58:10,948 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/528eb5f66cfd46b4aec10a4453ab1425, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/1c8bf2ff48734f41b9afcec9643dcbd2, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/6280e137855a46e0a7e3d6e8b8cca24a, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/0e0d17cef28d4b34b14c65c2dc7dad68, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/ad3380382bb74b76af2bce799fce724b, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/9108dcdb11a24828bc62023ddd9cbdee, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/6144753bd6004cf290290ef77a09e2b3, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/d7956de298824282b9fc6907ce674152, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/120261db92b2459bb056731a4b4d876c, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/82fdae4e85814080b564d772f54f322f, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/d8e570d7a24b4ffd93e51fce18000d40, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/5b8960c11f32419ab53052f0d9badc4b, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/8091f18f87c946fe9058db4c1abec559, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/558e8a5b750e4a22aeb14d75282c4414, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/6e7d181a6dbf4cd2b2fdad77eb9fb4a9, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/95eb8580931b40c0ab87bf00698a0a9b, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/1d436fb6cbc14a15a05faa18f645df94, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/3f72997246bc48d39cbd20781a4ad868, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/3fd1c30ffbe641b6b9e5d15e6872550a, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/012d92c768804da18254698936817ae2, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/b7e9e9fdd5a9415f850692f73c2d5e15, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/e6a675f6432f4952976fe3a4a007e084, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/a5db877b79664c97a148a4251ce113bf] to archive 2023-04-05 04:58:10,949 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 04:58:10,950 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/528eb5f66cfd46b4aec10a4453ab1425 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/528eb5f66cfd46b4aec10a4453ab1425 2023-04-05 04:58:10,951 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/1c8bf2ff48734f41b9afcec9643dcbd2 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/1c8bf2ff48734f41b9afcec9643dcbd2 2023-04-05 04:58:10,952 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/6280e137855a46e0a7e3d6e8b8cca24a to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/6280e137855a46e0a7e3d6e8b8cca24a 2023-04-05 04:58:10,953 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/0e0d17cef28d4b34b14c65c2dc7dad68 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/0e0d17cef28d4b34b14c65c2dc7dad68 2023-04-05 04:58:10,954 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/ad3380382bb74b76af2bce799fce724b to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/ad3380382bb74b76af2bce799fce724b 2023-04-05 04:58:10,955 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/9108dcdb11a24828bc62023ddd9cbdee to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/9108dcdb11a24828bc62023ddd9cbdee 2023-04-05 04:58:10,956 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/6144753bd6004cf290290ef77a09e2b3 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/6144753bd6004cf290290ef77a09e2b3 2023-04-05 04:58:10,957 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/d7956de298824282b9fc6907ce674152 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/d7956de298824282b9fc6907ce674152 2023-04-05 04:58:10,958 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/120261db92b2459bb056731a4b4d876c to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/120261db92b2459bb056731a4b4d876c 2023-04-05 04:58:10,962 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/82fdae4e85814080b564d772f54f322f to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/82fdae4e85814080b564d772f54f322f 2023-04-05 04:58:10,964 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/d8e570d7a24b4ffd93e51fce18000d40 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/d8e570d7a24b4ffd93e51fce18000d40 2023-04-05 04:58:10,964 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/5b8960c11f32419ab53052f0d9badc4b to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/5b8960c11f32419ab53052f0d9badc4b 2023-04-05 04:58:10,969 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/8091f18f87c946fe9058db4c1abec559 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/8091f18f87c946fe9058db4c1abec559 2023-04-05 04:58:10,970 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/558e8a5b750e4a22aeb14d75282c4414 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/558e8a5b750e4a22aeb14d75282c4414 2023-04-05 04:58:10,971 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/6e7d181a6dbf4cd2b2fdad77eb9fb4a9 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/6e7d181a6dbf4cd2b2fdad77eb9fb4a9 2023-04-05 04:58:10,973 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/95eb8580931b40c0ab87bf00698a0a9b to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/95eb8580931b40c0ab87bf00698a0a9b 2023-04-05 04:58:10,974 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/1d436fb6cbc14a15a05faa18f645df94 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/1d436fb6cbc14a15a05faa18f645df94 2023-04-05 04:58:10,975 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/3f72997246bc48d39cbd20781a4ad868 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/3f72997246bc48d39cbd20781a4ad868 2023-04-05 04:58:10,976 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/3fd1c30ffbe641b6b9e5d15e6872550a to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/3fd1c30ffbe641b6b9e5d15e6872550a 2023-04-05 04:58:10,977 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/012d92c768804da18254698936817ae2 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/012d92c768804da18254698936817ae2 2023-04-05 04:58:10,978 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/b7e9e9fdd5a9415f850692f73c2d5e15 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/b7e9e9fdd5a9415f850692f73c2d5e15 2023-04-05 04:58:10,979 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/e6a675f6432f4952976fe3a4a007e084 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/e6a675f6432f4952976fe3a4a007e084 2023-04-05 04:58:10,980 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/a5db877b79664c97a148a4251ce113bf to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/a5db877b79664c97a148a4251ce113bf 2023-04-05 04:58:10,986 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/33faa2f9280b46fe9f74f012a6d4fadc, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/a97c9e1c033a49f78909610b29b17e43, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/014a77a107964d94a4a26a9846973ac0, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/6de5715de2d140098a4afb6c4084062b, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/df0ba5c4dcfa4f19ac13577db8d10007, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/8821044e6c7648c2a3c87ea8ac08ab66, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/dbdb8dc4e0cd46ceb6f88d2c495e9fba, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/dcdeac5064d344b6a7c01eb2997a4a73, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/65ff6bca970e4b508418c71892c40861, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/73649c85cb3f44de9fc3d8931b201f48, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/e1897cf0d0a6465db040b8a34876a854, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/24d9708f1b0149b98f9e097bb4154932, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/22ebd28e1e4941218bdbf04b5c91ed77, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/59285e545fe44dfcaf8bb19b56deb184, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/95898a397f9640618c637a5601336bb3, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/6650fa4fdb0c48c8ab06cab41b0957ed, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/1d6863f5fa91410e9c67a3e969297d63, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/d9fb0cf85f884d57980a7092bcbc512f, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/dd443fd6f0554ccaa25aa8f305ec4e58, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/48448622064e4b269545dabcc40fa32f, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/bbd262734bcd454696df8e2e94177135, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/caadd1da5943447dbb87e2b04bac9354, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/f01daa9c541f43ba97535e4bd52f72c6] to archive 2023-04-05 04:58:10,986 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 04:58:10,988 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/33faa2f9280b46fe9f74f012a6d4fadc to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/33faa2f9280b46fe9f74f012a6d4fadc 2023-04-05 04:58:10,989 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/a97c9e1c033a49f78909610b29b17e43 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/a97c9e1c033a49f78909610b29b17e43 2023-04-05 04:58:10,990 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/014a77a107964d94a4a26a9846973ac0 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/014a77a107964d94a4a26a9846973ac0 2023-04-05 04:58:10,992 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/6de5715de2d140098a4afb6c4084062b to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/6de5715de2d140098a4afb6c4084062b 2023-04-05 04:58:10,993 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/df0ba5c4dcfa4f19ac13577db8d10007 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/df0ba5c4dcfa4f19ac13577db8d10007 2023-04-05 04:58:10,994 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/8821044e6c7648c2a3c87ea8ac08ab66 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/8821044e6c7648c2a3c87ea8ac08ab66 2023-04-05 04:58:10,996 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/dbdb8dc4e0cd46ceb6f88d2c495e9fba to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/dbdb8dc4e0cd46ceb6f88d2c495e9fba 2023-04-05 04:58:10,998 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/dcdeac5064d344b6a7c01eb2997a4a73 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/dcdeac5064d344b6a7c01eb2997a4a73 2023-04-05 04:58:10,999 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/65ff6bca970e4b508418c71892c40861 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/65ff6bca970e4b508418c71892c40861 2023-04-05 04:58:11,000 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/73649c85cb3f44de9fc3d8931b201f48 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/73649c85cb3f44de9fc3d8931b201f48 2023-04-05 04:58:11,001 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/e1897cf0d0a6465db040b8a34876a854 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/e1897cf0d0a6465db040b8a34876a854 2023-04-05 04:58:11,003 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/24d9708f1b0149b98f9e097bb4154932 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/24d9708f1b0149b98f9e097bb4154932 2023-04-05 04:58:11,005 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/22ebd28e1e4941218bdbf04b5c91ed77 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/22ebd28e1e4941218bdbf04b5c91ed77 2023-04-05 04:58:11,006 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/59285e545fe44dfcaf8bb19b56deb184 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/59285e545fe44dfcaf8bb19b56deb184 2023-04-05 04:58:11,007 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/95898a397f9640618c637a5601336bb3 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/95898a397f9640618c637a5601336bb3 2023-04-05 04:58:11,009 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/6650fa4fdb0c48c8ab06cab41b0957ed to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/6650fa4fdb0c48c8ab06cab41b0957ed 2023-04-05 04:58:11,010 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/1d6863f5fa91410e9c67a3e969297d63 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/1d6863f5fa91410e9c67a3e969297d63 2023-04-05 04:58:11,011 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/d9fb0cf85f884d57980a7092bcbc512f to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/d9fb0cf85f884d57980a7092bcbc512f 2023-04-05 04:58:11,012 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/dd443fd6f0554ccaa25aa8f305ec4e58 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/dd443fd6f0554ccaa25aa8f305ec4e58 2023-04-05 04:58:11,013 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/48448622064e4b269545dabcc40fa32f to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/48448622064e4b269545dabcc40fa32f 2023-04-05 04:58:11,015 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/bbd262734bcd454696df8e2e94177135 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/bbd262734bcd454696df8e2e94177135 2023-04-05 04:58:11,016 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/caadd1da5943447dbb87e2b04bac9354 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/caadd1da5943447dbb87e2b04bac9354 2023-04-05 04:58:11,017 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/f01daa9c541f43ba97535e4bd52f72c6 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/f01daa9c541f43ba97535e4bd52f72c6 2023-04-05 04:58:11,022 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/0cac5d39573c4344b6718fcb29b77c34, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/6a559d5bfa4f4a949f4463fe00bb09b3, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/533eca10fbbe42e39d786d2fd1ff00c1, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/adbf54ec6fdb45acbd92fdb0ad316369, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/b99cd8ec1571482a8c643e3a298d548e, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/75e7b6ff42ab4113b4e2e6205999bb40, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/a39e6337dbf743799871a1ff705cea3b, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/aea24ad35b8c45369808515755e0280a, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/759fdd54a2414a189db708d2040da14b, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/749447674c724ffc9b84e40292912534, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/6c55c83d84d14ce194684ce3736bb1cb, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/6fea3ce1ff1c4ad8a334b125f8c98fbb, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/7da440b195724986b95c74e2da4ee3d7, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/fc7441fe6c0c4f4dbfef4b2483217522, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/bb252bc8b06249038e5820912d8cf363, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/36095155b883478cacfe3b688ef2ecd3, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/c996e97d8ac7481c90356b3e2d51401a, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/609b773b588c4c798ee20b3b17ca8755, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/2a38924bed9a418890d451d18e512af4, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/36d4882e95c44a55abe238831c94e74c, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/767d832afe8f4829914a414b397bd9f2, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/3f960165605843679b6d8d5da520b8ba, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/52c7d1513a39427fa09c2f8e02f409e0] to archive 2023-04-05 04:58:11,023 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 04:58:11,024 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/0cac5d39573c4344b6718fcb29b77c34 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/0cac5d39573c4344b6718fcb29b77c34 2023-04-05 04:58:11,025 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/6a559d5bfa4f4a949f4463fe00bb09b3 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/6a559d5bfa4f4a949f4463fe00bb09b3 2023-04-05 04:58:11,026 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/533eca10fbbe42e39d786d2fd1ff00c1 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/533eca10fbbe42e39d786d2fd1ff00c1 2023-04-05 04:58:11,028 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/adbf54ec6fdb45acbd92fdb0ad316369 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/adbf54ec6fdb45acbd92fdb0ad316369 2023-04-05 04:58:11,029 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/b99cd8ec1571482a8c643e3a298d548e to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/b99cd8ec1571482a8c643e3a298d548e 2023-04-05 04:58:11,029 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/75e7b6ff42ab4113b4e2e6205999bb40 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/75e7b6ff42ab4113b4e2e6205999bb40 2023-04-05 04:58:11,031 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/a39e6337dbf743799871a1ff705cea3b to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/a39e6337dbf743799871a1ff705cea3b 2023-04-05 04:58:11,032 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/aea24ad35b8c45369808515755e0280a to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/aea24ad35b8c45369808515755e0280a 2023-04-05 04:58:11,033 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/759fdd54a2414a189db708d2040da14b to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/759fdd54a2414a189db708d2040da14b 2023-04-05 04:58:11,033 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/749447674c724ffc9b84e40292912534 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/749447674c724ffc9b84e40292912534 2023-04-05 04:58:11,034 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/6c55c83d84d14ce194684ce3736bb1cb to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/6c55c83d84d14ce194684ce3736bb1cb 2023-04-05 04:58:11,035 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/6fea3ce1ff1c4ad8a334b125f8c98fbb to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/6fea3ce1ff1c4ad8a334b125f8c98fbb 2023-04-05 04:58:11,036 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/7da440b195724986b95c74e2da4ee3d7 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/7da440b195724986b95c74e2da4ee3d7 2023-04-05 04:58:11,036 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/fc7441fe6c0c4f4dbfef4b2483217522 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/fc7441fe6c0c4f4dbfef4b2483217522 2023-04-05 04:58:11,037 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/bb252bc8b06249038e5820912d8cf363 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/bb252bc8b06249038e5820912d8cf363 2023-04-05 04:58:11,038 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/36095155b883478cacfe3b688ef2ecd3 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/36095155b883478cacfe3b688ef2ecd3 2023-04-05 04:58:11,039 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/c996e97d8ac7481c90356b3e2d51401a to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/c996e97d8ac7481c90356b3e2d51401a 2023-04-05 04:58:11,040 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/609b773b588c4c798ee20b3b17ca8755 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/609b773b588c4c798ee20b3b17ca8755 2023-04-05 04:58:11,041 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/2a38924bed9a418890d451d18e512af4 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/2a38924bed9a418890d451d18e512af4 2023-04-05 04:58:11,042 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/36d4882e95c44a55abe238831c94e74c to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/36d4882e95c44a55abe238831c94e74c 2023-04-05 04:58:11,043 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/767d832afe8f4829914a414b397bd9f2 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/767d832afe8f4829914a414b397bd9f2 2023-04-05 04:58:11,044 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/3f960165605843679b6d8d5da520b8ba to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/3f960165605843679b6d8d5da520b8ba 2023-04-05 04:58:11,045 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/52c7d1513a39427fa09c2f8e02f409e0 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/52c7d1513a39427fa09c2f8e02f409e0 2023-04-05 04:58:11,331 DEBUG [Thread-2228] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x74a355de to 127.0.0.1:56229 2023-04-05 04:58:11,331 DEBUG [Thread-2228] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:58:11,331 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-05 04:58:11,331 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(394): wrote 55 2023-04-05 04:58:11,331 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(394): wrote 47 2023-04-05 04:58:11,331 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(394): wrote 74 2023-04-05 04:58:11,332 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(394): wrote 72 2023-04-05 04:58:11,332 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(394): wrote 86 2023-04-05 04:58:11,332 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-05 04:58:11,332 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-05 04:58:11,332 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(402): scanned 1840 2023-04-05 04:58:11,332 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(403): verified 5519 rows 2023-04-05 04:58:11,332 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(402): scanned 1821 2023-04-05 04:58:11,332 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(403): verified 5462 rows 2023-04-05 04:58:11,332 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(402): scanned 1807 2023-04-05 04:58:11,332 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(403): verified 5417 rows 2023-04-05 04:58:11,332 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(402): scanned 1809 2023-04-05 04:58:11,332 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(403): verified 5423 rows 2023-04-05 04:58:11,332 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(402): scanned 1818 2023-04-05 04:58:11,332 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(403): verified 5450 rows 2023-04-05 04:58:11,332 INFO [Listener at localhost.localdomain/34579] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-05 04:58:11,332 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7d243e46 to 127.0.0.1:56229 2023-04-05 04:58:11,333 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:58:11,334 INFO [Listener at localhost.localdomain/34579] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-05 04:58:11,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-05 04:58:11,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-05 04:58:11,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-05 04:58:11,338 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680670691338"}]},"ts":"1680670691338"} 2023-04-05 04:58:11,339 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-05 04:58:11,344 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-05 04:58:11,345 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5708f0f53456b130f41c37899ae0e2e0, UNASSIGN}] 2023-04-05 04:58:11,346 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5708f0f53456b130f41c37899ae0e2e0, UNASSIGN 2023-04-05 04:58:11,347 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=5708f0f53456b130f41c37899ae0e2e0, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:11,347 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670691347"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680670691347"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680670691347"}]},"ts":"1680670691347"} 2023-04-05 04:58:11,348 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure 5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206}] 2023-04-05 04:58:11,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-05 04:58:11,500 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:58:11,500 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 5708f0f53456b130f41c37899ae0e2e0, disabling compactions & flushes 2023-04-05 04:58:11,500 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:58:11,500 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:58:11,500 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. after waiting 0 ms 2023-04-05 04:58:11,500 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:58:11,500 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 5708f0f53456b130f41c37899ae0e2e0 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-05 04:58:11,500 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=A 2023-04-05 04:58:11,500 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:11,500 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=B 2023-04-05 04:58:11,500 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:11,501 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5708f0f53456b130f41c37899ae0e2e0, store=C 2023-04-05 04:58:11,501 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:11,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-05 04:58:11,921 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405c459f9156fe245f6a362e03e445f5ee3 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405c459f9156fe245f6a362e03e445f5ee3 2023-04-05 04:58:11,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-05 04:58:12,328 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=413, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/298981484f6e49b5bb1b4f3fd25f6871 2023-04-05 04:58:12,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-05 04:58:12,746 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/6841bd58481048d0a797d945411bd3bd 2023-04-05 04:58:13,158 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/7bcfc88207a54a3db1eec472eabb142a 2023-04-05 04:58:13,162 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/A/298981484f6e49b5bb1b4f3fd25f6871 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/298981484f6e49b5bb1b4f3fd25f6871 2023-04-05 04:58:13,166 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/298981484f6e49b5bb1b4f3fd25f6871, entries=100, sequenceid=413, filesize=18.6 K 2023-04-05 04:58:13,167 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/B/6841bd58481048d0a797d945411bd3bd as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/6841bd58481048d0a797d945411bd3bd 2023-04-05 04:58:13,170 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/6841bd58481048d0a797d945411bd3bd, entries=100, sequenceid=413, filesize=9.5 K 2023-04-05 04:58:13,170 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/.tmp/C/7bcfc88207a54a3db1eec472eabb142a as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/7bcfc88207a54a3db1eec472eabb142a 2023-04-05 04:58:13,173 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/7bcfc88207a54a3db1eec472eabb142a, entries=100, sequenceid=413, filesize=9.5 K 2023-04-05 04:58:13,174 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 5708f0f53456b130f41c37899ae0e2e0 in 1674ms, sequenceid=413, compaction requested=false 2023-04-05 04:58:13,181 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/recovered.edits/416.seqid, newMaxSeqId=416, maxSeqId=4 2023-04-05 04:58:13,182 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0. 2023-04-05 04:58:13,182 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 5708f0f53456b130f41c37899ae0e2e0: 2023-04-05 04:58:13,183 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:58:13,184 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=5708f0f53456b130f41c37899ae0e2e0, regionState=CLOSED 2023-04-05 04:58:13,184 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670693184"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680670693184"}]},"ts":"1680670693184"} 2023-04-05 04:58:13,186 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-05 04:58:13,186 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure 5708f0f53456b130f41c37899ae0e2e0, server=jenkins-hbase20.apache.org,38091,1680670569206 in 1.8370 sec 2023-04-05 04:58:13,187 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-05 04:58:13,187 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=5708f0f53456b130f41c37899ae0e2e0, UNASSIGN in 1.8420 sec 2023-04-05 04:58:13,188 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680670693188"}]},"ts":"1680670693188"} 2023-04-05 04:58:13,189 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-05 04:58:13,189 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-05 04:58:13,191 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.8560 sec 2023-04-05 04:58:13,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-05 04:58:13,443 INFO [Listener at localhost.localdomain/34579] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-05 04:58:13,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-05 04:58:13,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 04:58:13,449 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 04:58:13,450 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 04:58:13,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-05 04:58:13,457 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:58:13,460 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A, FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B, FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C, FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/recovered.edits] 2023-04-05 04:58:13,464 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/298981484f6e49b5bb1b4f3fd25f6871 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/298981484f6e49b5bb1b4f3fd25f6871 2023-04-05 04:58:13,465 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/df9f10d497734953917db7749aec0135 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/A/df9f10d497734953917db7749aec0135 2023-04-05 04:58:13,467 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/6841bd58481048d0a797d945411bd3bd to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/6841bd58481048d0a797d945411bd3bd 2023-04-05 04:58:13,469 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/f9755e36b0604e389803d975bd8d31fb to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/B/f9755e36b0604e389803d975bd8d31fb 2023-04-05 04:58:13,471 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/7bcfc88207a54a3db1eec472eabb142a to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/7bcfc88207a54a3db1eec472eabb142a 2023-04-05 04:58:13,472 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/d8f313b841404ed7aaec828dbd8b4be4 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/C/d8f313b841404ed7aaec828dbd8b4be4 2023-04-05 04:58:13,474 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/recovered.edits/416.seqid to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0/recovered.edits/416.seqid 2023-04-05 04:58:13,475 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/5708f0f53456b130f41c37899ae0e2e0 2023-04-05 04:58:13,475 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-05 04:58:13,475 DEBUG [PEWorker-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-05 04:58:13,476 DEBUG [PEWorker-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-05 04:58:13,480 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040507cc19e22e66415083755992b27c8e5e to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040507cc19e22e66415083755992b27c8e5e 2023-04-05 04:58:13,481 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304051ca91d51f2724604957a459684c55944 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304051ca91d51f2724604957a459684c55944 2023-04-05 04:58:13,482 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304051cce739474824c288a10d3b8f754f02d to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304051cce739474824c288a10d3b8f754f02d 2023-04-05 04:58:13,483 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040521101bc38d344f7d848da1ceb65f1229 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040521101bc38d344f7d848da1ceb65f1229 2023-04-05 04:58:13,484 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304052d813e4338f44ff5b0da21493d95dae6 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304052d813e4338f44ff5b0da21493d95dae6 2023-04-05 04:58:13,484 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304053b2d54aefc4a444d9561397237e6b152 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304053b2d54aefc4a444d9561397237e6b152 2023-04-05 04:58:13,485 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304053d6be64e99764849a504083e3cc72e7a to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304053d6be64e99764849a504083e3cc72e7a 2023-04-05 04:58:13,486 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304054b28fa3726014c2ca19d8ad49649e68d to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304054b28fa3726014c2ca19d8ad49649e68d 2023-04-05 04:58:13,487 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405556aba58165f40bfbe1cafa95aa6aed7 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405556aba58165f40bfbe1cafa95aa6aed7 2023-04-05 04:58:13,488 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304055788813340c24ab9b668dbde9020be2e to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304055788813340c24ab9b668dbde9020be2e 2023-04-05 04:58:13,488 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304056a76542a5b1e436b9a11fae2c090ffe2 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304056a76542a5b1e436b9a11fae2c090ffe2 2023-04-05 04:58:13,489 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304056eb9e87a2af54c3aa2586c4e88c1aa0f to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304056eb9e87a2af54c3aa2586c4e88c1aa0f 2023-04-05 04:58:13,490 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304058bf46da8b1aa43138b307792c41010f8 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304058bf46da8b1aa43138b307792c41010f8 2023-04-05 04:58:13,491 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405a3f87894e56d48448f53ae81ced1e2c6 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405a3f87894e56d48448f53ae81ced1e2c6 2023-04-05 04:58:13,492 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405aef59b91d37b46f0bdb22abca4f931b7 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405aef59b91d37b46f0bdb22abca4f931b7 2023-04-05 04:58:13,493 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405b000c6c7623642a8945b3a72203627b6 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405b000c6c7623642a8945b3a72203627b6 2023-04-05 04:58:13,494 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405bbf41759205f419b8335e0fde99e9818 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405bbf41759205f419b8335e0fde99e9818 2023-04-05 04:58:13,495 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405c42e2a67575047ef89a927abc3859cdd to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405c42e2a67575047ef89a927abc3859cdd 2023-04-05 04:58:13,495 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405c459f9156fe245f6a362e03e445f5ee3 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405c459f9156fe245f6a362e03e445f5ee3 2023-04-05 04:58:13,496 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405e93195ed5a9a4c6daa88960fcd2c85c3 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405e93195ed5a9a4c6daa88960fcd2c85c3 2023-04-05 04:58:13,496 DEBUG [PEWorker-2] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-05 04:58:13,497 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 04:58:13,498 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-05 04:58:13,500 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-05 04:58:13,501 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 04:58:13,501 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-05 04:58:13,501 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680670693501"}]},"ts":"9223372036854775807"} 2023-04-05 04:58:13,502 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-05 04:58:13,502 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 5708f0f53456b130f41c37899ae0e2e0, NAME => 'TestAcidGuarantees,,1680670658315.5708f0f53456b130f41c37899ae0e2e0.', STARTKEY => '', ENDKEY => ''}] 2023-04-05 04:58:13,502 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-05 04:58:13,502 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680670693502"}]},"ts":"9223372036854775807"} 2023-04-05 04:58:13,503 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-05 04:58:13,505 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 04:58:13,506 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 60 msec 2023-04-05 04:58:13,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-05 04:58:13,552 INFO [Listener at localhost.localdomain/34579] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-05 04:58:13,564 INFO [Listener at localhost.localdomain/34579] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=229 (was 228) - Thread LEAK? -, OpenFileDescriptor=506 (was 539), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=504 (was 482) - SystemLoadAverage LEAK? -, ProcessCount=166 (was 166), AvailableMemoryMB=7179 (was 6336) - AvailableMemoryMB LEAK? - 2023-04-05 04:58:13,574 INFO [Listener at localhost.localdomain/34579] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=229, OpenFileDescriptor=506, MaxFileDescriptor=60000, SystemLoadAverage=504, ProcessCount=166, AvailableMemoryMB=7178 2023-04-05 04:58:13,576 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-05 04:58:13,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-05 04:58:13,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-05 04:58:13,578 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-05 04:58:13,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-05 04:58:13,579 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-05 04:58:13,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-05 04:58:13,580 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1 2023-04-05 04:58:13,580 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1 empty. 2023-04-05 04:58:13,581 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1 2023-04-05 04:58:13,581 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-05 04:58:13,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-05 04:58:13,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-05 04:58:13,991 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-05 04:58:13,993 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 3fee67ac8cf8f50b334cb2fd8fafdea1, NAME => 'TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp 2023-04-05 04:58:14,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-05 04:58:14,403 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 04:58:14,403 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 3fee67ac8cf8f50b334cb2fd8fafdea1, disabling compactions & flushes 2023-04-05 04:58:14,403 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:14,403 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:14,403 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. after waiting 0 ms 2023-04-05 04:58:14,403 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:14,403 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:14,403 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:14,405 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-05 04:58:14,407 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670694407"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680670694407"}]},"ts":"1680670694407"} 2023-04-05 04:58:14,408 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-05 04:58:14,408 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-05 04:58:14,409 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680670694409"}]},"ts":"1680670694409"} 2023-04-05 04:58:14,409 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-05 04:58:14,412 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3fee67ac8cf8f50b334cb2fd8fafdea1, ASSIGN}] 2023-04-05 04:58:14,413 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3fee67ac8cf8f50b334cb2fd8fafdea1, ASSIGN 2023-04-05 04:58:14,414 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=3fee67ac8cf8f50b334cb2fd8fafdea1, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,38091,1680670569206; forceNewPlan=false, retain=false 2023-04-05 04:58:14,566 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=3fee67ac8cf8f50b334cb2fd8fafdea1, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:14,566 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670694566"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680670694566"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680670694566"}]},"ts":"1680670694566"} 2023-04-05 04:58:14,569 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206}] 2023-04-05 04:58:14,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-05 04:58:14,724 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:14,724 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 3fee67ac8cf8f50b334cb2fd8fafdea1, NAME => 'TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.', STARTKEY => '', ENDKEY => ''} 2023-04-05 04:58:14,725 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 3fee67ac8cf8f50b334cb2fd8fafdea1 2023-04-05 04:58:14,725 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 04:58:14,725 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 3fee67ac8cf8f50b334cb2fd8fafdea1 2023-04-05 04:58:14,725 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 3fee67ac8cf8f50b334cb2fd8fafdea1 2023-04-05 04:58:14,726 INFO [StoreOpener-3fee67ac8cf8f50b334cb2fd8fafdea1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 3fee67ac8cf8f50b334cb2fd8fafdea1 2023-04-05 04:58:14,728 DEBUG [StoreOpener-3fee67ac8cf8f50b334cb2fd8fafdea1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A 2023-04-05 04:58:14,728 DEBUG [StoreOpener-3fee67ac8cf8f50b334cb2fd8fafdea1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A 2023-04-05 04:58:14,728 INFO [StoreOpener-3fee67ac8cf8f50b334cb2fd8fafdea1-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 04:58:14,729 INFO [StoreOpener-3fee67ac8cf8f50b334cb2fd8fafdea1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3fee67ac8cf8f50b334cb2fd8fafdea1 columnFamilyName A 2023-04-05 04:58:14,729 INFO [StoreOpener-3fee67ac8cf8f50b334cb2fd8fafdea1-1] regionserver.HStore(310): Store=3fee67ac8cf8f50b334cb2fd8fafdea1/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 04:58:14,729 INFO [StoreOpener-3fee67ac8cf8f50b334cb2fd8fafdea1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 3fee67ac8cf8f50b334cb2fd8fafdea1 2023-04-05 04:58:14,732 DEBUG [StoreOpener-3fee67ac8cf8f50b334cb2fd8fafdea1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B 2023-04-05 04:58:14,732 DEBUG [StoreOpener-3fee67ac8cf8f50b334cb2fd8fafdea1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B 2023-04-05 04:58:14,732 INFO [StoreOpener-3fee67ac8cf8f50b334cb2fd8fafdea1-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 04:58:14,732 INFO [StoreOpener-3fee67ac8cf8f50b334cb2fd8fafdea1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3fee67ac8cf8f50b334cb2fd8fafdea1 columnFamilyName B 2023-04-05 04:58:14,733 INFO [StoreOpener-3fee67ac8cf8f50b334cb2fd8fafdea1-1] regionserver.HStore(310): Store=3fee67ac8cf8f50b334cb2fd8fafdea1/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 04:58:14,733 INFO [StoreOpener-3fee67ac8cf8f50b334cb2fd8fafdea1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 3fee67ac8cf8f50b334cb2fd8fafdea1 2023-04-05 04:58:14,734 DEBUG [StoreOpener-3fee67ac8cf8f50b334cb2fd8fafdea1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C 2023-04-05 04:58:14,734 DEBUG [StoreOpener-3fee67ac8cf8f50b334cb2fd8fafdea1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C 2023-04-05 04:58:14,735 INFO [StoreOpener-3fee67ac8cf8f50b334cb2fd8fafdea1-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 04:58:14,735 INFO [StoreOpener-3fee67ac8cf8f50b334cb2fd8fafdea1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3fee67ac8cf8f50b334cb2fd8fafdea1 columnFamilyName C 2023-04-05 04:58:14,735 INFO [StoreOpener-3fee67ac8cf8f50b334cb2fd8fafdea1-1] regionserver.HStore(310): Store=3fee67ac8cf8f50b334cb2fd8fafdea1/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 04:58:14,736 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:14,736 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1 2023-04-05 04:58:14,737 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1 2023-04-05 04:58:14,739 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-05 04:58:14,740 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 3fee67ac8cf8f50b334cb2fd8fafdea1 2023-04-05 04:58:14,742 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-05 04:58:14,743 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 3fee67ac8cf8f50b334cb2fd8fafdea1; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74463092, jitterRate=0.10958653688430786}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-05 04:58:14,743 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:14,743 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1., pid=49, masterSystemTime=1680670694722 2023-04-05 04:58:14,745 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:14,745 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:14,745 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=3fee67ac8cf8f50b334cb2fd8fafdea1, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:14,745 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670694745"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680670694745"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680670694745"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680670694745"}]},"ts":"1680670694745"} 2023-04-05 04:58:14,749 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-05 04:58:14,749 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 in 178 msec 2023-04-05 04:58:14,750 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-05 04:58:14,750 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=3fee67ac8cf8f50b334cb2fd8fafdea1, ASSIGN in 337 msec 2023-04-05 04:58:14,751 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-05 04:58:14,751 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680670694751"}]},"ts":"1680670694751"} 2023-04-05 04:58:14,752 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-05 04:58:14,753 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-05 04:58:14,755 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1780 sec 2023-04-05 04:58:15,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-05 04:58:15,688 INFO [Listener at localhost.localdomain/34579] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-05 04:58:15,688 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x51ff7cd4 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:58:15,692 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2e8c6f6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:58:15,694 DEBUG [Listener at localhost.localdomain/34579] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:58:15,697 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:59280, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:58:15,698 DEBUG [Listener at localhost.localdomain/34579] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-05 04:58:15,701 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54540, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-05 04:58:15,702 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x06846250 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:58:15,708 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2cb7a788, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:58:15,708 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x647f95f6 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:58:15,714 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@41ccedcb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:58:15,715 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x58ae18e2 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:58:15,722 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@40e6f624, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:58:15,722 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x029c1e76 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:58:15,735 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2a5d7fcf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:58:15,736 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x2c240146 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:58:15,744 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4ea9d51a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:58:15,744 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x31fd1c1f to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:58:15,756 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@31fc0e3f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:58:15,757 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x209faa3c to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:58:15,764 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@79db57d4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:58:15,765 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x5345748a to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:58:15,776 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@e56ea94, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:58:15,777 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x3c9f61fa to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:58:15,785 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4486b3c7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:58:15,786 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x534826c2 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:58:15,797 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@149f1de0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:58:15,801 DEBUG [hconnection-0x5858828f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:58:15,803 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:59290, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:58:15,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:15,803 DEBUG [hconnection-0x72701385-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:58:15,803 DEBUG [hconnection-0x43c6c715-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:58:15,804 DEBUG [hconnection-0x2f6b086b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:58:15,804 DEBUG [hconnection-0x48764c98-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:58:15,806 DEBUG [hconnection-0x181cfc4c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:58:15,806 DEBUG [hconnection-0x4f60716a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:58:15,807 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:59304, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:58:15,807 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:59330, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:58:15,808 DEBUG [hconnection-0x1771ce82-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:58:15,808 DEBUG [hconnection-0x32255eb6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:58:15,809 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:59366, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:58:15,814 DEBUG [hconnection-0x1f34c701-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:58:15,817 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:59316, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:58:15,817 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:59314, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:58:15,817 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:59352, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:58:15,818 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:59340, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:58:15,818 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:59368, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:58:15,819 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:59384, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:58:15,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:15,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:15,822 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:15,822 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:15,822 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:15,822 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:15,824 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:15,824 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:15,825 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:15,825 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:15,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:15,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:15,826 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:15,826 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:15,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:15,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:15,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:15,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:15,827 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:15,828 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:15,828 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:15,829 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:15,829 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:15,829 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:15,829 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:15,829 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:15,829 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:15,829 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:15,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 3fee67ac8cf8f50b334cb2fd8fafdea1 2023-04-05 04:58:15,830 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3fee67ac8cf8f50b334cb2fd8fafdea1 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 04:58:15,830 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:15,831 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=A 2023-04-05 04:58:15,832 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:15,832 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=B 2023-04-05 04:58:15,832 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:15,832 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=C 2023-04-05 04:58:15,832 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:15,847 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/8351a6d639b141f8bdd8d7aa8c367ca3 2023-04-05 04:58:15,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:15,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59352 deadline: 1680670755847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:15,856 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:15,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59330 deadline: 1680670755848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:15,858 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:15,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59384 deadline: 1680670755853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:15,858 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:15,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59314 deadline: 1680670755853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:15,859 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:15,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59304 deadline: 1680670755854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:15,876 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/12ce1bf78742479d9fd63ffdf97aa0ef 2023-04-05 04:58:15,959 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:15,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59352 deadline: 1680670755956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:15,959 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:15,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59330 deadline: 1680670755956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:15,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:15,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59384 deadline: 1680670755959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:15,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:15,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59304 deadline: 1680670755960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:15,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:15,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59314 deadline: 1680670755960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:16,163 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:16,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59352 deadline: 1680670756160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:16,163 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:16,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59330 deadline: 1680670756161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:16,165 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:16,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59384 deadline: 1680670756163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:16,165 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:16,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59304 deadline: 1680670756163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:16,165 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:16,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59314 deadline: 1680670756163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:16,290 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/a2cff84b7368459e92516058c26763b7 2023-04-05 04:58:16,294 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/8351a6d639b141f8bdd8d7aa8c367ca3 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/8351a6d639b141f8bdd8d7aa8c367ca3 2023-04-05 04:58:16,297 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/8351a6d639b141f8bdd8d7aa8c367ca3, entries=250, sequenceid=14, filesize=16.2 K 2023-04-05 04:58:16,297 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/12ce1bf78742479d9fd63ffdf97aa0ef as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/12ce1bf78742479d9fd63ffdf97aa0ef 2023-04-05 04:58:16,301 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/12ce1bf78742479d9fd63ffdf97aa0ef, entries=150, sequenceid=14, filesize=11.6 K 2023-04-05 04:58:16,302 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/a2cff84b7368459e92516058c26763b7 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/a2cff84b7368459e92516058c26763b7 2023-04-05 04:58:16,308 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/a2cff84b7368459e92516058c26763b7, entries=150, sequenceid=14, filesize=11.6 K 2023-04-05 04:58:16,309 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 3fee67ac8cf8f50b334cb2fd8fafdea1 in 478ms, sequenceid=14, compaction requested=false 2023-04-05 04:58:16,309 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:16,309 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:16,309 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:16,309 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:16,309 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:16,309 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:16,309 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:16,314 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:16,314 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:16,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:16,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:16,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:16,315 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:16,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:16,315 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:16,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:16,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:16,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:16,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:16,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:16,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6958377[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:16,316 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:16,316 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:16,317 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:16,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:16,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:16,317 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:16,317 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:16,317 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:16,317 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:16,317 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:16,318 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:16,318 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:16,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:16,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:16,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:16,318 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:16,318 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:16,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:16,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:16,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:16,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:16,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:16,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:16,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:16,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:16,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:16,333 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:16,333 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:16,333 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:16,333 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:16,334 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:16,334 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:16,334 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:16,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:16,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:16,334 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:16,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:16,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:16,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:16,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:16,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:16,335 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:16,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:16,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:16,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:16,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:16,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:16,336 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:16,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:16,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:16,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:16,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:16,337 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:16,337 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:16,337 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:16,337 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:16,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:16,337 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:16,337 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:16,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:16,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:16,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:16,337 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:16,337 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:16,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:16,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:16,338 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:16,338 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:16,338 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:16,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:16,338 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:16,438 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:16,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:16,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:16,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:16,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:16,442 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:16,442 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:16,442 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:16,442 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:16,442 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:16,442 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:16,443 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:16,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:16,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:16,443 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:16,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:16,443 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:16,443 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:16,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:16,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:16,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:16,444 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:16,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:16,444 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:16,444 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:16,444 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:16,444 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:16,444 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:16,444 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:16,444 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:16,444 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3fee67ac8cf8f50b334cb2fd8fafdea1 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-05 04:58:16,445 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=A 2023-04-05 04:58:16,445 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:16,445 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=B 2023-04-05 04:58:16,445 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:16,445 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=C 2023-04-05 04:58:16,445 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:16,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 3fee67ac8cf8f50b334cb2fd8fafdea1 2023-04-05 04:58:16,466 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:16,475 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:16,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59384 deadline: 1680670756472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:16,479 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:16,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59330 deadline: 1680670756473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:16,479 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:16,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59314 deadline: 1680670756473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:16,479 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:16,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59304 deadline: 1680670756474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:16,481 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:16,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59352 deadline: 1680670756475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:16,577 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:16,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59384 deadline: 1680670756576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:16,581 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:16,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59330 deadline: 1680670756580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:16,582 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:16,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59314 deadline: 1680670756580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:16,582 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:16,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59304 deadline: 1680670756580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:16,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:16,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59352 deadline: 1680670756582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:16,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:16,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59384 deadline: 1680670756779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:16,785 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:16,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59314 deadline: 1680670756782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:16,785 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:16,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59330 deadline: 1680670756783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:16,785 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:16,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59304 deadline: 1680670756783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:16,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:16,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59352 deadline: 1680670756786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:16,850 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/26874a3134bc41fd8a4844a727fdf444 2023-04-05 04:58:17,085 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:17,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59384 deadline: 1680670757083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:17,089 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:17,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59330 deadline: 1680670757086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:17,089 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:17,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59304 deadline: 1680670757086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:17,090 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:17,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59314 deadline: 1680670757088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:17,090 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:17,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59352 deadline: 1680670757090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:17,259 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/cd54f3d3bbab4691bbe134a1fd964178 2023-04-05 04:58:17,266 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/da5cfbbfd19f4b678964847f1f7e98d4 2023-04-05 04:58:17,269 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/26874a3134bc41fd8a4844a727fdf444 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/26874a3134bc41fd8a4844a727fdf444 2023-04-05 04:58:17,271 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/26874a3134bc41fd8a4844a727fdf444, entries=150, sequenceid=34, filesize=11.6 K 2023-04-05 04:58:17,271 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/cd54f3d3bbab4691bbe134a1fd964178 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/cd54f3d3bbab4691bbe134a1fd964178 2023-04-05 04:58:17,274 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/cd54f3d3bbab4691bbe134a1fd964178, entries=150, sequenceid=34, filesize=11.6 K 2023-04-05 04:58:17,275 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/da5cfbbfd19f4b678964847f1f7e98d4 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/da5cfbbfd19f4b678964847f1f7e98d4 2023-04-05 04:58:17,277 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/da5cfbbfd19f4b678964847f1f7e98d4, entries=150, sequenceid=34, filesize=11.6 K 2023-04-05 04:58:17,278 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 3fee67ac8cf8f50b334cb2fd8fafdea1 in 834ms, sequenceid=34, compaction requested=false 2023-04-05 04:58:17,278 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:17,278 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:17,278 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:17,278 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:17,278 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:17,278 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:17,278 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:17,280 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:17,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:17,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:17,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:17,280 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:17,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:17,280 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:17,280 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:17,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:17,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:17,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:17,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:17,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:17,283 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:17,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@281a0926[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:17,283 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:17,283 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:17,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:17,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:17,283 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:17,283 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:17,283 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:17,284 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:17,284 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:17,284 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:17,284 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:17,284 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:17,284 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:17,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:17,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:17,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:17,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:17,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:17,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:17,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:17,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:17,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:17,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:17,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:17,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:17,288 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:17,288 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:17,288 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:17,288 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:17,288 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:17,292 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:17,292 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:17,292 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:17,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:17,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:17,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:17,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:17,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:17,293 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:17,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:17,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:17,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:17,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:17,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:17,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:17,294 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:17,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:17,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:17,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:17,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:17,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:17,295 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:17,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:17,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:17,295 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:17,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:17,295 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:17,296 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:17,296 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:17,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:17,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:17,296 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:17,296 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:17,296 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:17,296 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:17,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:17,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:17,296 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:17,296 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:17,296 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:17,395 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:17,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:17,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:17,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:17,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:17,398 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:17,398 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:17,398 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:17,398 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:17,399 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:17,399 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:17,401 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:17,401 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:17,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:17,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:17,402 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:17,402 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:17,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:17,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:17,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:17,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:17,402 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:17,402 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:17,402 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:17,402 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:17,402 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:17,402 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:17,402 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:17,403 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:17,403 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:17,403 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:17,403 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3fee67ac8cf8f50b334cb2fd8fafdea1 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-05 04:58:17,403 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=A 2023-04-05 04:58:17,403 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:17,403 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=B 2023-04-05 04:58:17,403 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:17,403 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=C 2023-04-05 04:58:17,403 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:17,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 3fee67ac8cf8f50b334cb2fd8fafdea1 2023-04-05 04:58:17,591 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:17,614 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:17,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59330 deadline: 1680670757605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:17,614 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:17,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59304 deadline: 1680670757606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:17,614 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:17,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59352 deadline: 1680670757607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:17,615 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:17,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59384 deadline: 1680670757608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:17,621 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:17,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59314 deadline: 1680670757614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:17,718 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:17,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59330 deadline: 1680670757715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:17,718 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:17,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59304 deadline: 1680670757715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:17,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:17,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59352 deadline: 1680670757715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:17,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:17,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59384 deadline: 1680670757716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:17,724 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:17,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59314 deadline: 1680670757722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:17,810 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/64073bfed5dc4123b223be53700b4d89 2023-04-05 04:58:17,921 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:17,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59304 deadline: 1680670757919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:17,924 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:17,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59330 deadline: 1680670757920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:17,924 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:17,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59352 deadline: 1680670757920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:17,925 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:17,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59384 deadline: 1680670757921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:17,926 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:17,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59314 deadline: 1680670757925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,217 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/77998cef7137475d9501dc1d9eeb790e 2023-04-05 04:58:18,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:18,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59304 deadline: 1680670758224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:18,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59330 deadline: 1680670758225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,233 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:18,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59352 deadline: 1680670758227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,233 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:18,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59384 deadline: 1680670758227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,234 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:18,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59314 deadline: 1680670758228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,625 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/cb68610214104ce7ba717da45d86b648 2023-04-05 04:58:18,628 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/64073bfed5dc4123b223be53700b4d89 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/64073bfed5dc4123b223be53700b4d89 2023-04-05 04:58:18,632 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/64073bfed5dc4123b223be53700b4d89, entries=150, sequenceid=46, filesize=11.6 K 2023-04-05 04:58:18,632 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-05 04:58:18,635 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/77998cef7137475d9501dc1d9eeb790e as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/77998cef7137475d9501dc1d9eeb790e 2023-04-05 04:58:18,638 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/77998cef7137475d9501dc1d9eeb790e, entries=150, sequenceid=46, filesize=11.6 K 2023-04-05 04:58:18,639 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/cb68610214104ce7ba717da45d86b648 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/cb68610214104ce7ba717da45d86b648 2023-04-05 04:58:18,641 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/cb68610214104ce7ba717da45d86b648, entries=150, sequenceid=46, filesize=11.6 K 2023-04-05 04:58:18,642 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 3fee67ac8cf8f50b334cb2fd8fafdea1 in 1239ms, sequenceid=46, compaction requested=true 2023-04-05 04:58:18,642 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:18,642 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:18,642 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:18,642 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:18,642 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,642 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:18,642 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:18,701 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,701 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:18,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:18,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:18,702 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:18,702 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:18,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:18,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:18,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:18,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:18,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:18,703 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:18,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1978649f[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:18,703 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:18,708 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:18,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:18,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:18,708 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,708 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:18,708 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:18,708 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:18,708 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:18,709 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:18,710 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,710 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:18,710 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:18,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:18,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:18,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:18,710 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-05 04:58:18,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:18,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:18,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:18,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:18,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:18,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:18,712 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:18,712 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:18,712 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:18,712 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:18,712 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:18,713 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:18,713 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:18,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:18,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:18,713 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:18,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:18,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:18,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:18,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:18,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:18,713 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:18,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:18,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:18,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:18,716 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:18,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:18,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:18,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,719 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:18,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:18,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:18,719 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:18,719 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:18,719 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,719 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:18,719 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:18,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:18,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:18,719 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:18,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:18,719 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:18,719 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,719 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:18,719 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:18,719 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:18,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:18,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:18,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 3fee67ac8cf8f50b334cb2fd8fafdea1 2023-04-05 04:58:18,737 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3fee67ac8cf8f50b334cb2fd8fafdea1 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-05 04:58:18,737 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=A 2023-04-05 04:58:18,737 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:58:18,737 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=B 2023-04-05 04:58:18,737 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:58:18,738 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=C 2023-04-05 04:58:18,738 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:58:18,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:18,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59304 deadline: 1680670758747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,758 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:18,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59330 deadline: 1680670758748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:18,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59314 deadline: 1680670758753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:18,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59352 deadline: 1680670758753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:18,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59384 deadline: 1680670758754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,819 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:18,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:18,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:18,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:18,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:18,822 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:18,822 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:18,822 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:18,822 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:18,823 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:18,823 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:18,823 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:18,823 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:18,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:18,824 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,824 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:18,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:18,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:18,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:18,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:18,824 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:18,830 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:18,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:18,830 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:18,830 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:18,830 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:18,830 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:18,830 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:18,830 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:18,830 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:18,831 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:18,831 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:18,831 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:18,831 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:18,831 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,831 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:18,831 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:18,832 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:18,832 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,832 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:18,832 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:18,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:18,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:18,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:18,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:18,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:18,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:18,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:18,837 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:18,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ad9341a[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:18,837 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:18,838 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:18,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:18,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:18,838 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,838 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:18,838 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:18,838 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:18,838 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:18,839 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,839 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:18,839 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:18,839 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:18,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:18,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:18,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:18,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:18,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:18,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:18,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:18,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:18,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:18,841 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:18,841 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:18,841 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:18,841 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:18,841 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:18,842 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:18,842 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:18,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:18,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:18,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:18,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:18,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:18,842 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:18,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:18,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:18,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:18,843 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:18,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:18,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:18,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:18,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,843 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:18,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:18,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,845 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:18,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:18,845 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:18,845 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:18,845 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:18,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:18,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:18,845 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:18,845 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:18,845 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,845 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:18,845 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:18,845 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:18,845 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:18,845 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:18,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:18,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:18,857 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:18,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59304 deadline: 1680670758855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,862 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:18,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59330 deadline: 1680670758859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,862 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:18,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59314 deadline: 1680670758860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:18,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59352 deadline: 1680670758860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,864 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:18,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59384 deadline: 1680670758860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,945 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:18,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:18,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:18,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:18,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:18,948 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:18,948 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:18,948 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:18,948 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:18,948 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:18,949 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:18,949 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:18,949 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:18,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:18,949 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,949 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:18,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:18,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:18,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:18,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:18,950 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:18,950 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:18,950 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:18,950 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:18,950 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:18,950 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:18,950 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:18,950 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:18,950 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:18,951 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:18,951 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:18,951 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:18,951 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:18,951 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:18,951 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,951 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:18,951 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:18,951 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,951 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:18,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:18,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:18,952 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:18,952 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:18,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:18,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:18,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:18,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:18,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:18,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@507d8e0d[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:18,953 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:18,953 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:18,953 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,953 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:18,954 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:18,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:18,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:18,954 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:18,954 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:18,954 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:18,955 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,955 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:18,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:18,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:18,956 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:18,956 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:18,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:18,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:18,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:18,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:18,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:18,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:18,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:18,957 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:18,957 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:18,957 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:18,957 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:18,957 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:18,958 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:18,958 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:18,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:18,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:18,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:18,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:18,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:18,958 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:18,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:18,958 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:18,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:18,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:18,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:18,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:18,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:18,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,960 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:18,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:18,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,961 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,961 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:18,961 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:18,961 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:18,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:18,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:18,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:18,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:18,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:18,961 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:18,961 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:18,961 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:18,961 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:18,961 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:18,961 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:18,961 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:18,961 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:18,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:18,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:19,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:19,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59304 deadline: 1680670759058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,061 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:19,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:19,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:19,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:19,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:19,064 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:19,064 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:19,064 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:19,064 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:19,065 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,065 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:19,065 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,065 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:19,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:19,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:19,065 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,065 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:19,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:19,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:19,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:19,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59330 deadline: 1680670759064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:19,066 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:19,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:19,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59314 deadline: 1680670759064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,066 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:19,066 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:19,067 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:19,067 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:19,067 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:19,067 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:19,067 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:19,067 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:19,067 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:19,067 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:19,067 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:19,067 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:19,067 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:19,067 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,067 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:19,067 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:19,068 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,068 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:19,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:19,068 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:19,068 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:19,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:19,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:19,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:19,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:19,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28d6ee1d[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:19,069 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:19,069 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,069 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,070 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:19,070 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:19,070 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,070 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:19,070 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:19,070 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:19,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59352 deadline: 1680670759066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,070 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:19,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59384 deadline: 1680670759066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,071 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,071 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:19,071 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:19,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:19,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:19,071 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:19,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:19,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:19,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:19,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:19,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:19,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:19,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:19,072 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:19,072 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:19,072 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:19,073 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:19,073 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:19,073 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,073 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:19,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,073 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:19,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:19,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:19,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:19,074 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:19,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:19,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:19,075 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:19,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:19,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:19,076 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,076 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:19,076 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:19,076 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:19,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:19,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:19,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:19,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:19,076 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:19,076 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:19,077 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:19,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:19,077 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:19,077 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,077 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,077 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,077 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:19,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:19,147 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/53e3b5735d2c48f6b5a29ad4566e8f8a 2023-04-05 04:58:19,177 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:19,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:19,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:19,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:19,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:19,180 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:19,180 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:19,180 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:19,180 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:19,180 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,180 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:19,181 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,181 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:19,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:19,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:19,181 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,181 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:19,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:19,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:19,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:19,182 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:19,182 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:19,182 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:19,182 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:19,182 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:19,182 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:19,182 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:19,182 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:19,182 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:19,183 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:19,183 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:19,183 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:19,183 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:19,183 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:19,183 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,183 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:19,183 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:19,184 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,184 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:19,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:19,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:19,185 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:19,185 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:19,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:19,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:19,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:19,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72503b04[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:19,186 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:19,186 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,187 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,187 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:19,187 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,187 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:19,187 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:19,187 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:19,188 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,188 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:19,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:19,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:19,188 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:19,188 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:19,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:19,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:19,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:19,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:19,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:19,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:19,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:19,190 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:19,190 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:19,190 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:19,190 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:19,190 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:19,191 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:19,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:19,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:19,191 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,191 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:19,191 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:19,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:19,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:19,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:19,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:19,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,192 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:19,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:19,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,194 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,194 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:19,194 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:19,194 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:19,194 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:19,194 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,194 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:19,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:19,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:19,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:19,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:19,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:19,194 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,194 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,194 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,194 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:19,194 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:19,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:19,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:19,294 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:19,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:19,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:19,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:19,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:19,297 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:19,297 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:19,297 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:19,297 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:19,298 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,298 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:19,298 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:19,298 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:19,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:19,299 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,299 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:19,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:19,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:19,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:19,299 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:19,299 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:19,299 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:19,300 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:19,300 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:19,300 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:19,300 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:19,300 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:19,300 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:19,300 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:19,300 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:19,300 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:19,300 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:19,300 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:19,300 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,300 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:19,300 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:19,301 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,301 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:19,301 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,301 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:19,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:19,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:19,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@419aa8b0[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:19,302 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:19,302 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,303 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,303 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:19,303 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:19,303 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:19,303 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,303 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:19,303 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:19,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:19,304 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:19,304 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:19,304 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:19,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:19,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:19,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:19,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:19,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:19,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:19,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:19,306 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:19,307 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:19,307 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:19,307 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:19,307 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:19,308 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,308 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,308 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:19,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:19,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:19,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:19,308 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:19,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:19,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:19,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:19,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:19,309 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:19,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:19,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,310 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,310 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:19,310 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:19,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:19,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:19,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:19,310 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:19,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:19,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:19,311 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:19,311 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:19,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:19,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:19,311 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,311 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:19,311 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,311 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:19,311 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,311 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,365 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:19,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59304 deadline: 1680670759363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,371 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:19,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59314 deadline: 1680670759368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,371 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:19,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59330 deadline: 1680670759369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:19,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59352 deadline: 1680670759372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,377 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:19,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59384 deadline: 1680670759373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,411 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:19,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:19,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:19,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:19,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:19,414 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:19,414 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:19,414 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:19,414 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:19,414 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,414 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:19,415 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,416 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:19,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:19,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:19,416 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,416 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:19,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:19,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:19,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:19,416 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:19,418 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:19,418 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:19,418 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:19,418 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:19,418 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:19,418 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:19,418 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:19,418 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:19,418 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:19,419 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:19,419 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:19,419 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:19,419 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:19,419 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,419 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:19,419 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:19,419 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,419 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:19,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:19,420 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,420 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:19,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:19,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:19,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:19,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:19,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:19,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d2b5b68[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:19,421 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:19,421 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,421 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,421 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,421 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:19,421 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:19,421 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:19,421 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:19,422 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,422 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:19,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:19,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:19,422 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:19,422 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:19,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:19,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:19,424 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:19,424 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:19,424 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:19,424 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:19,424 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:19,425 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,425 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:19,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:19,425 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:19,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:19,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:19,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:19,425 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:19,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:19,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:19,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,426 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:19,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:19,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,427 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,427 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:19,427 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:19,428 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:19,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:19,428 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:19,428 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:19,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:19,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:19,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:19,428 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,428 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:19,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:19,428 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:19,428 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,428 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:19,428 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,528 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:19,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:19,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:19,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:19,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:19,531 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:19,531 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:19,531 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:19,531 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:19,531 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,531 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:19,532 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,532 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:19,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:19,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:19,532 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,532 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:19,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:19,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:19,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:19,533 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:19,533 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:19,533 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:19,533 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:19,533 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:19,533 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:19,533 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:19,533 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:19,533 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:19,533 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:19,533 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:19,533 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:19,533 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:19,533 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:19,534 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,534 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:19,534 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:19,534 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,534 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:19,535 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,535 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:19,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:19,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6292f4d6[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:19,536 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:19,536 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,536 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,536 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:19,536 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:19,536 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:19,537 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,537 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:19,537 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,537 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:19,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:19,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:19,537 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:19,537 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:19,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:19,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:19,539 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:19,539 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:19,539 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:19,539 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:19,539 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:19,540 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,540 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:19,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,540 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:19,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:19,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:19,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:19,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,542 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:19,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:19,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:19,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,543 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:19,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:19,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,544 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,544 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:19,544 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:19,544 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:19,544 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,545 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:19,545 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:19,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:19,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:19,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:19,544 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:19,545 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:19,545 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:19,545 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:19,545 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:19,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:19,556 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/3761b9ba0d044dc0ae2e22f5756bfe6d 2023-04-05 04:58:19,565 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/532f666e5318497b91519b504d738e47 2023-04-05 04:58:19,570 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/53e3b5735d2c48f6b5a29ad4566e8f8a as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/53e3b5735d2c48f6b5a29ad4566e8f8a 2023-04-05 04:58:19,572 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/53e3b5735d2c48f6b5a29ad4566e8f8a, entries=200, sequenceid=67, filesize=13.9 K 2023-04-05 04:58:19,573 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/3761b9ba0d044dc0ae2e22f5756bfe6d as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/3761b9ba0d044dc0ae2e22f5756bfe6d 2023-04-05 04:58:19,579 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/3761b9ba0d044dc0ae2e22f5756bfe6d, entries=150, sequenceid=67, filesize=11.6 K 2023-04-05 04:58:19,580 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/532f666e5318497b91519b504d738e47 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/532f666e5318497b91519b504d738e47 2023-04-05 04:58:19,582 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/532f666e5318497b91519b504d738e47, entries=150, sequenceid=67, filesize=11.6 K 2023-04-05 04:58:19,583 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 3fee67ac8cf8f50b334cb2fd8fafdea1 in 846ms, sequenceid=67, compaction requested=true 2023-04-05 04:58:19,583 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:19,583 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:58:19,584 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:58:19,584 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:58:19,584 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:58:19,586 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:58:19,590 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:58:19,591 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): 3fee67ac8cf8f50b334cb2fd8fafdea1/B is initiating minor compaction (all files) 2023-04-05 04:58:19,591 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3fee67ac8cf8f50b334cb2fd8fafdea1/B in TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:19,591 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/12ce1bf78742479d9fd63ffdf97aa0ef, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/cd54f3d3bbab4691bbe134a1fd964178, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/77998cef7137475d9501dc1d9eeb790e, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/3761b9ba0d044dc0ae2e22f5756bfe6d] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp, totalSize=46.4 K 2023-04-05 04:58:19,591 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54564 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:58:19,591 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 12ce1bf78742479d9fd63ffdf97aa0ef, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680670695825 2023-04-05 04:58:19,591 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): 3fee67ac8cf8f50b334cb2fd8fafdea1/A is initiating minor compaction (all files) 2023-04-05 04:58:19,591 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3fee67ac8cf8f50b334cb2fd8fafdea1/A in TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:19,591 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/8351a6d639b141f8bdd8d7aa8c367ca3, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/26874a3134bc41fd8a4844a727fdf444, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/64073bfed5dc4123b223be53700b4d89, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/53e3b5735d2c48f6b5a29ad4566e8f8a] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp, totalSize=53.3 K 2023-04-05 04:58:19,591 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting cd54f3d3bbab4691bbe134a1fd964178, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1680670695845 2023-04-05 04:58:19,591 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 8351a6d639b141f8bdd8d7aa8c367ca3, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680670695825 2023-04-05 04:58:19,592 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 77998cef7137475d9501dc1d9eeb790e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1680670696471 2023-04-05 04:58:19,592 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 26874a3134bc41fd8a4844a727fdf444, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1680670695845 2023-04-05 04:58:19,593 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 3761b9ba0d044dc0ae2e22f5756bfe6d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1680670697604 2023-04-05 04:58:19,593 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 64073bfed5dc4123b223be53700b4d89, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1680670696471 2023-04-05 04:58:19,593 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 53e3b5735d2c48f6b5a29ad4566e8f8a, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1680670697604 2023-04-05 04:58:19,599 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3fee67ac8cf8f50b334cb2fd8fafdea1#B#compaction#374 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:58:19,601 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): 3fee67ac8cf8f50b334cb2fd8fafdea1#A#compaction#375 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:58:19,645 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:19,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:19,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:19,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:19,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:19,647 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:19,647 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:19,648 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:19,648 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:19,648 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,648 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:19,648 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:19,648 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:19,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:19,649 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,649 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:19,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:19,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:19,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:19,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:19,649 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:19,650 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:19,650 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:19,650 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:19,650 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:19,650 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:19,650 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:19,650 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:19,650 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:19,650 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:19,650 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3fee67ac8cf8f50b334cb2fd8fafdea1 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 04:58:19,650 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=A 2023-04-05 04:58:19,650 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:19,650 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=B 2023-04-05 04:58:19,651 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:19,651 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=C 2023-04-05 04:58:19,651 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:19,657 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=78 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/9440036470844f8e898f09b8987cfceb 2023-04-05 04:58:19,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 3fee67ac8cf8f50b334cb2fd8fafdea1 2023-04-05 04:58:19,872 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:19,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:19,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59304 deadline: 1680670759893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680670699883, max timestamp=1680670699898Num uniques 859; 2023-04-05 04:58:19,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670699883, max timestamp=1680670699898Num uniques 859; 2023-04-05 04:58:19,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680670699883, max timestamp=1680670699898Num uniques 859; 2023-04-05 04:58:19,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:58:19,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 04:58:19,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 04:58:19,928 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:19,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59314 deadline: 1680670759925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,933 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:19,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59352 deadline: 1680670759926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,935 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:19,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59384 deadline: 1680670759928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:19,935 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:19,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59330 deadline: 1680670759928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,003 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:20,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59304 deadline: 1680670760001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,012 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/ed20c2a5895b4352aef63971b49119b5 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/ed20c2a5895b4352aef63971b49119b5 2023-04-05 04:58:20,015 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3fee67ac8cf8f50b334cb2fd8fafdea1/B of 3fee67ac8cf8f50b334cb2fd8fafdea1 into ed20c2a5895b4352aef63971b49119b5(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:58:20,015 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:20,016 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1., storeName=3fee67ac8cf8f50b334cb2fd8fafdea1/B, priority=12, startTime=1680670699584; duration=0sec 2023-04-05 04:58:20,016 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:58:20,016 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:58:20,017 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:58:20,017 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): 3fee67ac8cf8f50b334cb2fd8fafdea1/C is initiating minor compaction (all files) 2023-04-05 04:58:20,017 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3fee67ac8cf8f50b334cb2fd8fafdea1/C in TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:20,017 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/a2cff84b7368459e92516058c26763b7, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/da5cfbbfd19f4b678964847f1f7e98d4, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/cb68610214104ce7ba717da45d86b648, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/532f666e5318497b91519b504d738e47] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp, totalSize=46.4 K 2023-04-05 04:58:20,017 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting a2cff84b7368459e92516058c26763b7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680670695825 2023-04-05 04:58:20,018 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting da5cfbbfd19f4b678964847f1f7e98d4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1680670695845 2023-04-05 04:58:20,018 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting cb68610214104ce7ba717da45d86b648, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1680670696471 2023-04-05 04:58:20,018 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 532f666e5318497b91519b504d738e47, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1680670697604 2023-04-05 04:58:20,021 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/4a561539dea94d39a456600a86fcc651 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/4a561539dea94d39a456600a86fcc651 2023-04-05 04:58:20,024 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3fee67ac8cf8f50b334cb2fd8fafdea1/A of 3fee67ac8cf8f50b334cb2fd8fafdea1 into 4a561539dea94d39a456600a86fcc651(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:58:20,024 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:20,024 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1., storeName=3fee67ac8cf8f50b334cb2fd8fafdea1/A, priority=12, startTime=1680670699583; duration=0sec 2023-04-05 04:58:20,024 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:58:20,024 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3fee67ac8cf8f50b334cb2fd8fafdea1#C#compaction#378 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:58:20,044 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:20,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59314 deadline: 1680670760030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,044 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:20,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59330 deadline: 1680670760036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,044 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:20,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59352 deadline: 1680670760038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,044 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:20,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59384 deadline: 1680670760038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,065 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=78 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/bc56df46434a49b8b9477f15dc38337c 2023-04-05 04:58:20,074 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=78 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/8e6400aff4ae4e1e8c13ee0358ef8cb9 2023-04-05 04:58:20,081 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/9440036470844f8e898f09b8987cfceb as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/9440036470844f8e898f09b8987cfceb 2023-04-05 04:58:20,084 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/9440036470844f8e898f09b8987cfceb, entries=150, sequenceid=78, filesize=11.6 K 2023-04-05 04:58:20,084 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/bc56df46434a49b8b9477f15dc38337c as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/bc56df46434a49b8b9477f15dc38337c 2023-04-05 04:58:20,092 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/bc56df46434a49b8b9477f15dc38337c, entries=150, sequenceid=78, filesize=11.6 K 2023-04-05 04:58:20,092 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/8e6400aff4ae4e1e8c13ee0358ef8cb9 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/8e6400aff4ae4e1e8c13ee0358ef8cb9 2023-04-05 04:58:20,099 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/8e6400aff4ae4e1e8c13ee0358ef8cb9, entries=150, sequenceid=78, filesize=11.6 K 2023-04-05 04:58:20,100 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=129.71 KB/132820 for 3fee67ac8cf8f50b334cb2fd8fafdea1 in 450ms, sequenceid=78, compaction requested=false 2023-04-05 04:58:20,100 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:20,100 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:20,100 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:20,100 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:20,100 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,100 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:20,100 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:20,101 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,101 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:20,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:20,101 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,101 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:20,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:20,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:20,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:20,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:20,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11e96b9b[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:20,102 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:20,102 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,103 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,103 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:20,103 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:20,103 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:20,103 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,103 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:20,103 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,103 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:20,103 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:20,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:20,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:20,103 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:20,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:20,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:20,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:20,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:20,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:20,105 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:20,105 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:20,105 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:20,105 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:20,105 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:20,106 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,106 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,106 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:20,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:20,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:20,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,106 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:20,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:20,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:20,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:20,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,107 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:20,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,109 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:20,109 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:20,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:20,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:20,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:20,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:20,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:20,116 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:20,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:20,116 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:20,116 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:20,116 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:20,116 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:20,116 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,116 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,116 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,116 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,116 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:20,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 3fee67ac8cf8f50b334cb2fd8fafdea1 2023-04-05 04:58:20,208 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3fee67ac8cf8f50b334cb2fd8fafdea1 3/3 column families, dataSize=136.42 KB heapSize=385.38 KB 2023-04-05 04:58:20,208 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=A 2023-04-05 04:58:20,208 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:58:20,208 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=B 2023-04-05 04:58:20,208 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:58:20,208 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=C 2023-04-05 04:58:20,208 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:58:20,217 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:20,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:20,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:20,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:20,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:20,219 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:20,219 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:20,219 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:20,219 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:20,219 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,220 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:20,220 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,220 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:20,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:20,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:20,220 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,220 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:20,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:20,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:20,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:20,221 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:20,222 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:20,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:20,222 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:20,222 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:20,222 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:20,224 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:20,224 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:20,224 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:20,224 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:20,224 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:20,224 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:20,224 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:20,224 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:20,224 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,224 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:20,224 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:20,225 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,225 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:20,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:20,225 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,225 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:20,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:20,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:20,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:20,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:20,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5228e3e[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:20,226 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:20,226 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,227 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,227 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:20,227 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:20,227 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:20,227 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,227 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:20,228 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,228 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:20,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:20,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:20,228 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:20,228 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:20,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:20,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:20,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:20,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:20,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:20,230 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:20,230 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:20,230 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:20,230 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:20,230 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:20,231 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,231 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,231 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:20,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:20,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:20,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:20,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:20,232 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:20,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:20,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,232 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:20,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,234 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,234 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:20,234 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:20,234 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:20,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:20,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:20,234 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:20,234 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,234 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,234 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:20,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:20,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:20,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:20,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:20,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:20,234 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,234 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:20,234 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,234 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:20,252 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:20,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59352 deadline: 1680670760246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,252 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:20,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59384 deadline: 1680670760246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,254 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:20,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59304 deadline: 1680670760248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:20,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59314 deadline: 1680670760251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:20,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59330 deadline: 1680670760252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,335 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:20,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:20,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:20,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:20,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:20,337 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:20,337 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:20,337 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:20,337 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:20,338 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,338 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:20,338 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:20,338 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:20,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:20,338 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,339 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:20,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:20,339 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:20,340 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:20,340 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:20,340 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:20,340 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:20,340 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:20,340 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:20,340 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:20,340 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:20,340 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:20,340 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:20,340 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:20,340 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:20,340 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:20,340 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,340 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:20,340 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:20,341 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:20,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:20,341 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:20,342 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,342 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:20,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:20,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22f575e9[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:20,343 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:20,343 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,343 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,343 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:20,343 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:20,343 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:20,343 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,343 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:20,344 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,344 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:20,344 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:20,344 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:20,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:20,348 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:20,348 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:20,348 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:20,348 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:20,348 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:20,349 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,349 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:20,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:20,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,349 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:20,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,350 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,351 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:20,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,352 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,352 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:20,352 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:20,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:20,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:20,352 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:20,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:20,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:20,352 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:20,352 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:20,352 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:20,352 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,353 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:20,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:20,352 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:20,353 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,353 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:20,353 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,357 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:20,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59304 deadline: 1680670760354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:20,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59314 deadline: 1680670760360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:20,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59330 deadline: 1680670760360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,437 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/c839b4861b614016811bf13a1e66ce73 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/c839b4861b614016811bf13a1e66ce73 2023-04-05 04:58:20,440 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3fee67ac8cf8f50b334cb2fd8fafdea1/C of 3fee67ac8cf8f50b334cb2fd8fafdea1 into c839b4861b614016811bf13a1e66ce73(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:58:20,440 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:20,440 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1., storeName=3fee67ac8cf8f50b334cb2fd8fafdea1/C, priority=12, startTime=1680670699586; duration=0sec 2023-04-05 04:58:20,441 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:58:20,453 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:20,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:20,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:20,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:20,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:20,455 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:20,455 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:20,455 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:20,456 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:20,456 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,456 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:20,456 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:20,456 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:20,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:20,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:20,457 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,457 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:20,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:20,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:20,458 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:20,458 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:20,458 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:20,458 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:20,458 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:20,458 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:20,458 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:20,458 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:20,458 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:20,458 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:20,458 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:20,458 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:20,458 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:20,458 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:20,458 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,459 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:20,459 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:20,459 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,459 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:20,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:20,460 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,460 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:20,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:20,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:20,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:20,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:20,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57341125[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:20,460 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:20,461 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,461 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,461 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,461 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:20,461 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:20,461 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:20,461 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:20,462 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,462 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:20,462 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:20,462 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:20,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:20,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:20,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:20,465 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:20,465 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:20,465 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:20,465 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:20,465 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:20,465 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,465 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:20,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,466 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:20,466 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:20,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,467 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:20,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,470 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:20,470 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,470 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:20,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:20,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:20,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:20,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:20,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:20,470 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:20,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:20,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:20,470 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:20,471 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:20,471 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:20,471 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:20,471 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,471 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,471 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,471 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,558 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:20,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59352 deadline: 1680670760555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,558 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:20,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59384 deadline: 1680670760555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,563 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:20,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59304 deadline: 1680670760559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,571 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:20,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:20,572 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:20,572 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:20,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59330 deadline: 1680670760565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59314 deadline: 1680670760566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:20,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:20,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:20,574 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:20,574 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:20,574 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:20,574 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:20,574 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,574 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:20,575 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,575 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:20,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:20,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:20,575 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,575 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:20,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:20,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:20,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:20,576 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:20,576 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:20,576 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:20,576 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:20,576 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:20,576 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:20,576 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:20,576 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:20,576 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:20,576 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:20,576 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:20,576 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:20,576 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:20,576 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:20,576 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,576 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:20,576 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:20,577 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:20,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:20,577 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:20,577 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,578 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:20,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:20,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d5b1deb[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:20,578 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:20,579 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,579 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,579 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:20,579 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:20,579 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:20,579 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,579 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:20,580 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:20,580 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,580 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:20,580 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:20,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:20,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:20,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:20,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:20,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:20,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:20,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:20,582 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:20,582 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:20,582 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:20,582 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:20,582 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:20,583 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,583 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:20,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:20,583 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:20,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:20,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:20,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:20,583 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:20,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,584 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:20,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,586 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,586 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:20,586 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:20,586 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:20,586 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:20,586 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:20,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:20,586 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:20,586 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:20,586 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:20,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:20,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:20,586 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,587 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,587 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:20,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:20,617 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/bfdf4408a323482d825b966f8af4c3ff 2023-04-05 04:58:20,686 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:20,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:20,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:20,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:20,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:20,689 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:20,689 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:20,689 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:20,689 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:20,689 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,689 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:20,690 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:20,690 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:20,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:20,690 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,690 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:20,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:20,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:20,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:20,691 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:20,691 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:20,691 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:20,691 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:20,691 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:20,691 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:20,691 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:20,691 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:20,691 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:20,691 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:20,691 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:20,692 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:20,692 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:20,692 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:20,692 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,692 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:20,692 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:20,692 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,692 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:20,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:20,693 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,693 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:20,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:20,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c992100[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:20,694 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:20,694 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,694 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,694 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:20,694 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:20,694 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:20,694 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,694 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:20,695 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,695 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:20,695 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:20,695 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:20,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:20,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:20,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:20,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:20,697 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:20,697 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:20,697 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:20,697 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:20,697 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:20,698 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,698 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:20,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:20,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,698 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:20,698 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:20,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:20,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:20,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:20,700 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:20,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,701 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,701 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:20,701 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:20,702 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,702 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:20,702 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:20,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:20,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:20,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:20,702 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:20,702 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:20,702 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:20,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:20,702 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,702 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:20,703 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:20,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:20,802 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:20,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:20,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:20,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:20,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:20,805 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:20,805 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:20,805 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:20,805 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:20,805 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,805 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:20,806 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,806 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:20,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:20,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:20,806 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,806 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:20,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:20,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:20,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:20,807 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:20,807 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:20,807 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:20,807 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:20,807 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:20,807 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:20,807 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:20,807 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:20,807 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:20,807 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:20,807 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:20,807 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:20,807 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:20,807 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:20,807 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,807 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:20,807 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:20,808 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,808 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:20,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:20,808 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,808 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:20,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:20,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:20,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:20,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:20,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15ca4547[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:20,809 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:20,810 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,810 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,810 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:20,810 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:20,810 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:20,810 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,810 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:20,811 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,811 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:20,811 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:20,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:20,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:20,811 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:20,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:20,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:20,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:20,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:20,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:20,813 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:20,813 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:20,813 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:20,813 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:20,813 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:20,814 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,814 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,814 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:20,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:20,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:20,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:20,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:20,814 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:20,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:20,815 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:20,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,816 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:20,816 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:20,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:20,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:20,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:20,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:20,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,831 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:20,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:20,832 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:20,831 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:20,832 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:20,832 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,832 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:20,832 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:20,832 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,832 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,832 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,832 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:20,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:20,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59304 deadline: 1680670760866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,876 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:20,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59314 deadline: 1680670760873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,879 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:20,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59330 deadline: 1680670760875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,932 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:20,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:20,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:20,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:20,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:20,935 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:20,935 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:20,935 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:20,935 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:20,935 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,935 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:20,936 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:20,936 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:20,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:20,936 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:20,936 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:20,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:20,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:20,937 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:20,937 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:20,937 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:20,937 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:20,937 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:20,937 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:20,937 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:20,937 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:20,937 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:20,937 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:20,938 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:20,938 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:20,938 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:20,938 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,938 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:20,938 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:20,939 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,939 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:20,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:20,939 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,940 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:20,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:20,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:20,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:20,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:20,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6280c3e2[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:20,941 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:20,941 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,941 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,941 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:20,941 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,941 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:20,941 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:20,941 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:20,942 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,942 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:20,942 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:20,942 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:20,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:20,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:20,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:20,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:20,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:20,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:20,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:20,944 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:20,944 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:20,944 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:20,944 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:20,944 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:20,945 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,945 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,945 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:20,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:20,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:20,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:20,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:20,946 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:20,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:20,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:20,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,947 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:20,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:20,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,949 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,949 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:20,949 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:20,949 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:20,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:20,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:20,949 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:20,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:20,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:20,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:20,949 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:20,949 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:20,949 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:20,949 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:20,950 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:20,950 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:20,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:20,950 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,026 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/b3ffd557fabb4d38bc4e64199a7151c3 2023-04-05 04:58:21,050 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:21,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:21,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:21,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:21,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:21,052 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:21,053 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:21,053 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:21,053 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:21,053 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,053 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:21,054 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:21,054 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:21,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:21,054 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,054 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:21,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:21,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:21,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:21,055 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:21,055 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:21,055 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:21,055 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:21,055 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:21,055 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:21,055 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:21,055 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:21,055 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:21,055 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:21,055 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:21,055 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:21,055 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:21,055 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:21,056 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,056 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:21,056 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:21,056 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,056 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:21,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:21,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:21,057 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,057 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:21,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:21,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:21,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:21,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:21,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37612e1c[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:21,058 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:21,058 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,058 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,058 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:21,058 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:21,058 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:21,059 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,059 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:21,059 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:21,059 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:21,059 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,059 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:21,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:21,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:21,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:21,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:21,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:21,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:21,061 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:21,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:21,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59352 deadline: 1680670761059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:21,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:21,062 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:21,062 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:21,062 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:21,062 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:21,062 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:21,063 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,063 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:21,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:21,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,063 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:21,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:21,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:21,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:21,063 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:21,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:21,064 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:21,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59384 deadline: 1680670761061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:21,064 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:21,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:21,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,066 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,066 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:21,066 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:21,066 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:21,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:21,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:21,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:21,066 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,066 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,066 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,066 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,066 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:21,066 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:21,068 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:21,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:21,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:21,068 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:21,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:21,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:21,166 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:21,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:21,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:21,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:21,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:21,169 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:21,169 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:21,169 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:21,169 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:21,169 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,169 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:21,170 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,170 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:21,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:21,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:21,170 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,170 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:21,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:21,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:21,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:21,171 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:21,171 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:21,171 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:21,171 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:21,171 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:21,171 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:21,172 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:21,172 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:21,172 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:21,172 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:21,172 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:21,172 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:21,172 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:21,172 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:21,172 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,172 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:21,172 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:21,173 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,173 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:21,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:21,173 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,173 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:21,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:21,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:21,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:21,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:21,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:21,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@622588d1[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:21,174 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:21,174 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,174 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,174 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:21,174 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:21,174 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,175 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:21,174 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:21,175 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:21,175 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:21,175 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,175 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:21,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:21,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:21,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:21,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:21,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:21,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:21,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:21,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:21,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:21,177 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:21,177 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:21,177 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:21,177 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:21,177 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:21,178 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,178 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:21,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,178 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:21,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:21,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:21,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:21,178 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:21,179 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:21,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:21,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,180 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,181 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:21,181 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:21,181 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:21,181 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:21,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:21,181 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:21,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:21,181 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:21,181 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,181 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,181 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:21,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:21,181 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:21,181 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:21,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:21,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:21,281 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:21,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:21,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:21,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:21,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:21,284 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:21,284 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:21,284 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:21,284 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:21,284 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,284 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:21,285 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:21,285 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:21,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:21,285 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,285 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:21,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:21,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:21,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:21,285 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:21,286 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:21,286 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:21,286 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:21,286 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:21,286 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:21,286 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:21,286 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:21,286 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:21,286 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:21,286 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:21,286 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:21,286 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:21,286 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:21,286 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,286 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:21,286 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:21,288 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,288 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,288 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,288 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:21,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:21,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:21,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:21,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:21,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:21,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:21,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:21,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@258219af[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:21,289 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:21,289 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,291 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,291 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:21,291 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:21,291 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:21,291 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,291 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:21,291 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:21,291 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,291 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:21,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:21,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:21,291 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:21,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:21,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:21,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:21,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:21,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:21,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:21,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:21,293 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:21,293 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:21,293 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:21,293 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:21,293 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:21,294 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,294 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:21,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:21,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,294 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:21,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:21,294 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,295 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:21,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:21,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,296 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,297 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:21,297 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:21,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:21,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:21,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:21,297 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:21,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:21,297 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:21,297 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:21,297 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:21,297 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:21,297 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,297 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:21,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:21,297 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,297 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,374 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:21,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59304 deadline: 1680670761371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,383 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:21,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59314 deadline: 1680670761378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,385 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:21,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59330 deadline: 1680670761382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,397 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:21,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:21,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:21,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:21,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:21,399 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:21,400 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:21,400 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:21,400 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:21,400 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,400 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:21,401 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,401 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:21,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:21,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:21,401 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,401 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:21,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:21,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:21,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:21,401 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:21,401 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:21,402 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:21,402 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:21,402 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:21,402 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:21,402 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:21,402 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:21,402 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:21,402 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:21,402 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:21,402 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:21,402 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:21,402 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:21,402 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,402 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:21,402 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:21,403 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,403 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:21,403 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:21,403 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:21,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:21,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:21,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:21,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41f9512e[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:21,404 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:21,404 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,405 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,405 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:21,405 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,405 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:21,405 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:21,405 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:21,406 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:21,406 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:21,406 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:21,406 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:21,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:21,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:21,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:21,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:21,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:21,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:21,408 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:21,408 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:21,408 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:21,408 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:21,408 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:21,409 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,409 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:21,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:21,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,409 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:21,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:21,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:21,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:21,410 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:21,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:21,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:21,411 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:21,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:21,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,412 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,412 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:21,412 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:21,412 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:21,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:21,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:21,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:21,412 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:21,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:21,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:21,412 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:21,413 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,413 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,413 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,413 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,413 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:21,413 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:21,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:21,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:21,436 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/ebc4752e67a743a885ad64933c4419c0 2023-04-05 04:58:21,440 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/bfdf4408a323482d825b966f8af4c3ff as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/bfdf4408a323482d825b966f8af4c3ff 2023-04-05 04:58:21,443 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/bfdf4408a323482d825b966f8af4c3ff, entries=150, sequenceid=116, filesize=11.6 K 2023-04-05 04:58:21,443 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/b3ffd557fabb4d38bc4e64199a7151c3 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/b3ffd557fabb4d38bc4e64199a7151c3 2023-04-05 04:58:21,449 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/b3ffd557fabb4d38bc4e64199a7151c3, entries=150, sequenceid=116, filesize=11.6 K 2023-04-05 04:58:21,449 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/ebc4752e67a743a885ad64933c4419c0 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/ebc4752e67a743a885ad64933c4419c0 2023-04-05 04:58:21,452 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/ebc4752e67a743a885ad64933c4419c0, entries=150, sequenceid=116, filesize=11.6 K 2023-04-05 04:58:21,453 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~136.42 KB/139690, heapSize ~385.33 KB/394576, currentSize=46.96 KB/48090 for 3fee67ac8cf8f50b334cb2fd8fafdea1 in 1245ms, sequenceid=116, compaction requested=true 2023-04-05 04:58:21,453 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:21,453 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:58:21,453 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:58:21,453 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:58:21,453 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:58:21,453 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:58:21,454 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:58:21,454 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:58:21,454 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): 3fee67ac8cf8f50b334cb2fd8fafdea1/A is initiating minor compaction (all files) 2023-04-05 04:58:21,454 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): 3fee67ac8cf8f50b334cb2fd8fafdea1/B is initiating minor compaction (all files) 2023-04-05 04:58:21,454 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3fee67ac8cf8f50b334cb2fd8fafdea1/A in TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:21,454 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3fee67ac8cf8f50b334cb2fd8fafdea1/B in TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:21,454 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/4a561539dea94d39a456600a86fcc651, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/9440036470844f8e898f09b8987cfceb, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/bfdf4408a323482d825b966f8af4c3ff] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp, totalSize=35.0 K 2023-04-05 04:58:21,454 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/ed20c2a5895b4352aef63971b49119b5, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/bc56df46434a49b8b9477f15dc38337c, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/b3ffd557fabb4d38bc4e64199a7151c3] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp, totalSize=35.0 K 2023-04-05 04:58:21,454 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 4a561539dea94d39a456600a86fcc651, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1680670697604 2023-04-05 04:58:21,454 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting ed20c2a5895b4352aef63971b49119b5, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1680670697604 2023-04-05 04:58:21,455 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 9440036470844f8e898f09b8987cfceb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=78, earliestPutTs=1680670698738 2023-04-05 04:58:21,455 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting bc56df46434a49b8b9477f15dc38337c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=78, earliestPutTs=1680670698738 2023-04-05 04:58:21,455 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting bfdf4408a323482d825b966f8af4c3ff, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1680670699924 2023-04-05 04:58:21,455 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting b3ffd557fabb4d38bc4e64199a7151c3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1680670699924 2023-04-05 04:58:21,469 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3fee67ac8cf8f50b334cb2fd8fafdea1#A#compaction#383 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:58:21,470 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): 3fee67ac8cf8f50b334cb2fd8fafdea1#B#compaction#384 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:58:21,483 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/6070a8e7582e4ae49a661844d7e36abb as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/6070a8e7582e4ae49a661844d7e36abb 2023-04-05 04:58:21,487 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3fee67ac8cf8f50b334cb2fd8fafdea1/B of 3fee67ac8cf8f50b334cb2fd8fafdea1 into 6070a8e7582e4ae49a661844d7e36abb(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:58:21,487 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:21,487 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1., storeName=3fee67ac8cf8f50b334cb2fd8fafdea1/B, priority=13, startTime=1680670701453; duration=0sec 2023-04-05 04:58:21,487 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:58:21,487 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:58:21,488 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:58:21,488 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): 3fee67ac8cf8f50b334cb2fd8fafdea1/C is initiating minor compaction (all files) 2023-04-05 04:58:21,488 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3fee67ac8cf8f50b334cb2fd8fafdea1/C in TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:21,488 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/c839b4861b614016811bf13a1e66ce73, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/8e6400aff4ae4e1e8c13ee0358ef8cb9, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/ebc4752e67a743a885ad64933c4419c0] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp, totalSize=35.0 K 2023-04-05 04:58:21,488 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting c839b4861b614016811bf13a1e66ce73, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1680670697604 2023-04-05 04:58:21,488 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 8e6400aff4ae4e1e8c13ee0358ef8cb9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=78, earliestPutTs=1680670698738 2023-04-05 04:58:21,489 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting ebc4752e67a743a885ad64933c4419c0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1680670699924 2023-04-05 04:58:21,493 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): 3fee67ac8cf8f50b334cb2fd8fafdea1#C#compaction#385 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:58:21,503 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/9f3f8407116f4ccca92318fe8cd9f321 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/9f3f8407116f4ccca92318fe8cd9f321 2023-04-05 04:58:21,509 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3fee67ac8cf8f50b334cb2fd8fafdea1/C of 3fee67ac8cf8f50b334cb2fd8fafdea1 into 9f3f8407116f4ccca92318fe8cd9f321(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:58:21,509 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:21,509 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1., storeName=3fee67ac8cf8f50b334cb2fd8fafdea1/C, priority=13, startTime=1680670701453; duration=0sec 2023-04-05 04:58:21,509 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:58:21,513 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:21,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:21,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:21,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:21,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:21,515 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:21,515 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:21,515 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:21,515 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:21,516 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,516 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:21,518 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:21,518 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:21,518 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,518 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:21,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:21,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:21,519 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:21,519 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:21,519 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:21,519 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:21,519 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:21,519 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:21,519 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:21,519 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:21,519 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:21,519 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:21,519 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3fee67ac8cf8f50b334cb2fd8fafdea1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 04:58:21,520 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=A 2023-04-05 04:58:21,520 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:21,520 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=B 2023-04-05 04:58:21,520 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:21,520 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=C 2023-04-05 04:58:21,520 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:21,528 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=129 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/687b79f194a84eaca138bb2e41122b0e 2023-04-05 04:58:21,538 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=129 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/7a0059aa79404f71ae4debec48313c34 2023-04-05 04:58:21,883 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/460a4fc48c494b2bae3bb5d171b8c7ac as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/460a4fc48c494b2bae3bb5d171b8c7ac 2023-04-05 04:58:21,886 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3fee67ac8cf8f50b334cb2fd8fafdea1/A of 3fee67ac8cf8f50b334cb2fd8fafdea1 into 460a4fc48c494b2bae3bb5d171b8c7ac(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:58:21,886 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:21,886 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1., storeName=3fee67ac8cf8f50b334cb2fd8fafdea1/A, priority=13, startTime=1680670701453; duration=0sec 2023-04-05 04:58:21,886 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:58:21,948 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=129 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/fe29fc5f4094489f9edbba46a5688220 2023-04-05 04:58:21,951 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/687b79f194a84eaca138bb2e41122b0e as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/687b79f194a84eaca138bb2e41122b0e 2023-04-05 04:58:21,953 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/687b79f194a84eaca138bb2e41122b0e, entries=100, sequenceid=129, filesize=9.3 K 2023-04-05 04:58:21,956 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/7a0059aa79404f71ae4debec48313c34 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/7a0059aa79404f71ae4debec48313c34 2023-04-05 04:58:21,959 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/7a0059aa79404f71ae4debec48313c34, entries=100, sequenceid=129, filesize=9.3 K 2023-04-05 04:58:21,963 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/fe29fc5f4094489f9edbba46a5688220 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/fe29fc5f4094489f9edbba46a5688220 2023-04-05 04:58:21,966 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/fe29fc5f4094489f9edbba46a5688220, entries=100, sequenceid=129, filesize=9.3 K 2023-04-05 04:58:21,967 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for 3fee67ac8cf8f50b334cb2fd8fafdea1 in 448ms, sequenceid=129, compaction requested=false 2023-04-05 04:58:21,967 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:21,967 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:21,967 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:21,967 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:21,967 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,968 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:21,968 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:21,972 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,972 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:21,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:21,972 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,973 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:21,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:21,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:21,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:21,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:21,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:21,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f5696c3[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:21,974 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:21,974 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,975 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,975 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:21,975 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:21,975 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,975 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:21,975 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:21,975 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:21,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:21,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:21,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:21,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:21,976 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:21,976 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:21,976 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:21,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:21,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:21,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:21,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:21,978 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:21,978 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:21,978 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:21,978 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:21,978 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:21,978 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,978 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:21,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,978 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:21,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:21,979 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:21,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:21,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,980 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:21,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:21,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,981 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:21,981 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:21,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:21,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:21,981 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,981 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:21,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:21,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:21,981 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:21,982 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:21,982 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,982 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:21,982 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:21,981 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:21,982 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:21,982 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:21,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:21,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:22,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 3fee67ac8cf8f50b334cb2fd8fafdea1 2023-04-05 04:58:22,079 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3fee67ac8cf8f50b334cb2fd8fafdea1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 04:58:22,079 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=A 2023-04-05 04:58:22,079 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:22,079 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=B 2023-04-05 04:58:22,079 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:22,079 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=C 2023-04-05 04:58:22,079 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:22,082 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:22,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:22,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:22,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:22,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:22,085 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:22,085 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:22,085 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:22,085 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:22,085 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,085 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:22,089 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:22,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:22,089 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:22,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:22,089 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,089 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:22,090 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=140 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/e42965417dd5439d88f8db83a2776640 2023-04-05 04:58:22,090 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:22,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:22,090 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:22,090 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:22,090 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:22,090 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:22,090 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:22,090 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:22,090 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:22,090 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:22,090 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:22,090 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:22,090 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:22,090 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:22,091 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:22,091 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:22,091 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,091 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:22,091 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:22,096 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,096 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:22,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:22,096 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,096 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:22,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:22,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:22,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:22,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:22,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@317bfaf8[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:22,098 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:22,098 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,101 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,101 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:22,101 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:22,101 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,101 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:22,101 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:22,101 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:22,101 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,101 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:22,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:22,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:22,102 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:22,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:22,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:22,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:22,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:22,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:22,103 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:22,103 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:22,103 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:22,103 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:22,103 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:22,105 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,105 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:22,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:22,105 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:22,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:22,106 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:22,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,106 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:22,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,107 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,107 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:22,107 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:22,107 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,107 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,107 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:22,107 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,108 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:22,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:22,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:22,108 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:22,108 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:22,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:22,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:22,113 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:22,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:22,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:22,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670702104, max timestamp=1680670702133Num uniques 859; 2023-04-05 04:58:22,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670702104, max timestamp=1680670702133Num uniques 859; 2023-04-05 04:58:22,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670702104, max timestamp=1680670702133Num uniques 859; 2023-04-05 04:58:22,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:58:22,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:58:22,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:58:22,187 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:22,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59352 deadline: 1680670762181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,190 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:22,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59384 deadline: 1680670762188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,208 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:22,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:22,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:22,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:22,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:22,210 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:22,210 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:22,210 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:22,211 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:22,211 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,211 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:22,211 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:22,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:22,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:22,211 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:22,212 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,212 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:22,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:22,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:22,212 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:22,213 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:22,213 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:22,213 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:22,213 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:22,213 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:22,213 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:22,213 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:22,213 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:22,213 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:22,213 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:22,213 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:22,213 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:22,213 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:22,213 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,213 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:22,213 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:22,214 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,214 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:22,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:22,215 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,215 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:22,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:22,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:22,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:22,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:22,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@401d219[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:22,216 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:22,216 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,216 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,216 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:22,216 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:22,216 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,216 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:22,216 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:22,217 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:22,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:22,217 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:22,217 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:22,217 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:22,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:22,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:22,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:22,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:22,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:22,219 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:22,219 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:22,219 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:22,219 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:22,219 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:22,220 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:22,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:22,220 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,220 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:22,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:22,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:22,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:22,221 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:22,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,222 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:22,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,223 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,223 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:22,223 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:22,223 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:22,223 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:22,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:22,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:22,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:22,223 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:22,223 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:22,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:22,223 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:22,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:22,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:22,223 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,224 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:22,224 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,224 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,293 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:22,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59352 deadline: 1680670762288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,294 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:22,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59384 deadline: 1680670762291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,324 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:22,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:22,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:22,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:22,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:22,326 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:22,326 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:22,326 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:22,326 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:22,326 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,326 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:22,327 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:22,327 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:22,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:22,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:22,327 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,327 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:22,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:22,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:22,328 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:22,328 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:22,328 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:22,328 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:22,328 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:22,328 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:22,328 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:22,328 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:22,328 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:22,328 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:22,328 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:22,329 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:22,329 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:22,329 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:22,329 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,329 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:22,329 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:22,329 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,329 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:22,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:22,330 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,330 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:22,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:22,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:22,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:22,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:22,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f7a8396[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:22,331 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:22,331 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,331 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,332 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:22,332 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:22,332 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:22,332 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,332 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:22,332 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:22,332 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,332 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:22,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:22,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:22,332 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:22,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:22,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:22,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:22,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:22,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:22,334 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:22,334 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:22,334 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:22,334 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:22,334 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:22,335 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,335 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:22,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:22,335 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:22,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:22,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:22,336 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:22,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:22,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,336 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:22,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,338 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,338 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:22,338 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:22,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:22,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:22,338 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:22,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:22,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:22,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:22,338 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:22,338 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:22,338 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:22,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:22,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:22,338 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:22,338 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,338 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,338 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,338 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:22,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59304 deadline: 1680670762376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,394 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:22,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59314 deadline: 1680670762389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,395 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:22,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59330 deadline: 1680670762392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,438 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:22,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:22,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:22,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:22,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:22,441 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:22,441 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:22,441 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:22,441 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:22,441 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,441 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:22,442 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:22,442 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:22,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:22,442 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,442 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:22,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:22,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:22,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:22,443 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:22,443 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:22,443 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:22,443 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:22,443 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:22,443 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:22,443 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:22,443 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:22,443 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:22,443 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:22,443 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:22,443 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:22,443 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:22,443 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:22,443 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,444 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:22,444 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:22,444 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,444 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:22,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:22,445 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,445 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:22,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:22,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:22,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:22,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:22,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4451ec34[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:22,446 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:22,446 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,446 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,446 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,446 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:22,446 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:22,446 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:22,446 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:22,447 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,447 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:22,447 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:22,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:22,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:22,447 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:22,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:22,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:22,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:22,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:22,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:22,449 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:22,449 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:22,449 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:22,449 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:22,449 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:22,450 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,450 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:22,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:22,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,450 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:22,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:22,450 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:22,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:22,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:22,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,451 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:22,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,453 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,453 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:22,453 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:22,453 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:22,453 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:22,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:22,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:22,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:22,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:22,453 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:22,453 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,453 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,453 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:22,454 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:22,453 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:22,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:22,454 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,495 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:22,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59352 deadline: 1680670762494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,498 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:22,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59384 deadline: 1680670762495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,498 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=140 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/6162d88224b74d399892061a4df62e49 2023-04-05 04:58:22,554 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:22,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:22,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:22,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:22,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:22,556 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:22,556 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:22,556 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:22,556 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:22,557 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,557 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:22,557 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:22,557 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:22,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:22,558 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,558 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:22,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:22,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:22,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:22,558 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:22,558 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:22,558 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:22,558 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:22,558 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:22,558 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:22,559 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:22,559 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:22,559 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:22,559 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:22,559 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:22,559 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:22,559 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:22,559 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:22,559 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,559 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:22,559 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:22,561 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,561 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:22,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:22,561 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,561 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:22,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:22,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:22,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:22,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:22,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a13627c[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:22,562 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:22,562 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,563 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,563 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:22,563 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:22,563 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:22,563 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,563 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:22,566 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,566 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:22,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:22,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:22,566 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:22,567 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:22,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:22,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:22,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:22,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:22,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:22,568 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:22,568 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:22,568 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:22,568 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:22,568 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:22,569 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,569 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,569 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:22,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:22,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:22,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:22,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:22,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:22,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:22,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,570 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:22,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,571 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,573 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,573 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:22,573 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:22,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:22,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:22,573 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,573 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:22,573 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,573 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,573 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:22,573 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,573 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:22,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:22,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:22,573 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:22,573 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:22,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:22,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:22,676 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:22,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:22,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:22,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:22,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:22,679 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:22,679 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:22,679 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:22,679 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:22,680 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,680 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:22,681 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,681 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:22,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:22,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:22,681 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,681 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:22,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:22,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:22,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:22,682 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:22,682 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:22,682 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:22,682 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:22,682 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:22,682 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:22,682 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:22,682 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:22,682 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:22,683 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:22,683 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:22,683 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:22,683 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:22,683 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:22,683 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,683 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:22,683 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:22,684 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,684 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:22,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:22,684 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,684 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:22,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:22,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:22,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:22,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:22,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2088ec3d[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:22,685 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:22,685 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,686 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,686 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:22,686 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:22,686 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:22,688 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,688 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:22,688 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,688 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:22,688 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:22,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:22,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:22,688 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:22,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:22,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:22,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:22,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:22,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:22,690 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:22,690 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:22,690 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:22,691 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:22,691 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:22,691 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,691 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,692 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:22,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:22,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:22,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:22,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:22,692 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:22,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:22,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,695 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:22,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,696 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,697 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:22,697 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:22,697 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:22,697 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:22,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:22,697 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:22,697 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:22,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:22,697 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,697 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,697 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:22,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:22,697 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:22,697 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:22,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:22,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:22,800 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:22,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:22,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:22,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:22,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:22,803 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:22,803 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:22,803 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:22,803 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:22,804 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,804 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:22,804 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,804 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:22,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:22,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:22,805 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,805 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:22,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:22,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,805 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:22,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59384 deadline: 1680670762801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:22,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:22,805 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:22,805 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:22,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59352 deadline: 1680670762801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,806 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:22,806 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:22,806 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:22,806 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:22,806 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:22,806 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:22,806 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:22,806 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:22,806 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:22,806 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:22,806 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:22,806 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:22,806 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,806 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:22,806 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:22,807 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:22,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:22,807 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:22,808 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,808 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:22,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:22,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:22,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:22,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b35bd89[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:22,809 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:22,809 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,809 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,810 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:22,810 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:22,810 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:22,810 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,810 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:22,811 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,811 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:22,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:22,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:22,811 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:22,811 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:22,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:22,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:22,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:22,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:22,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:22,813 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:22,813 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:22,813 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:22,813 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:22,813 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:22,814 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,814 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:22,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:22,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,814 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:22,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,814 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:22,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:22,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:22,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:22,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,815 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:22,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:22,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,817 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,817 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:22,817 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:22,817 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:22,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:22,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:22,817 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:22,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:22,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:22,817 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:22,817 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:22,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:22,817 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,817 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:22,818 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:22,818 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,818 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:22,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:22,906 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=140 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/c10ea1bc882c4df594285bc681dae059 2023-04-05 04:58:22,909 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/e42965417dd5439d88f8db83a2776640 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/e42965417dd5439d88f8db83a2776640 2023-04-05 04:58:22,911 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/e42965417dd5439d88f8db83a2776640, entries=200, sequenceid=140, filesize=14.1 K 2023-04-05 04:58:22,914 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/6162d88224b74d399892061a4df62e49 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/6162d88224b74d399892061a4df62e49 2023-04-05 04:58:22,918 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:22,918 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/6162d88224b74d399892061a4df62e49, entries=150, sequenceid=140, filesize=11.8 K 2023-04-05 04:58:22,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:22,919 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/c10ea1bc882c4df594285bc681dae059 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/c10ea1bc882c4df594285bc681dae059 2023-04-05 04:58:22,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:22,921 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/c10ea1bc882c4df594285bc681dae059, entries=150, sequenceid=140, filesize=11.8 K 2023-04-05 04:58:22,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:22,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:22,922 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:22,922 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:22,922 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:22,922 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:22,922 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 3fee67ac8cf8f50b334cb2fd8fafdea1 in 843ms, sequenceid=140, compaction requested=true 2023-04-05 04:58:22,922 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:22,922 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,922 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:22,922 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:58:22,922 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:58:22,923 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:58:22,923 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:58:22,923 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:58:22,923 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:22,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:22,923 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:22,923 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:22,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:22,923 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:22,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:22,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:22,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:22,924 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:22,924 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:22,924 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:22,924 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:22,924 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:22,924 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:22,925 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:22,925 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:22,925 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:22,926 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:22,926 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3fee67ac8cf8f50b334cb2fd8fafdea1 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-05 04:58:22,926 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36094 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:58:22,926 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=A 2023-04-05 04:58:22,926 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): 3fee67ac8cf8f50b334cb2fd8fafdea1/A is initiating minor compaction (all files) 2023-04-05 04:58:22,926 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:58:22,926 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3fee67ac8cf8f50b334cb2fd8fafdea1/A in TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:22,926 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=B 2023-04-05 04:58:22,926 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/460a4fc48c494b2bae3bb5d171b8c7ac, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/687b79f194a84eaca138bb2e41122b0e, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/e42965417dd5439d88f8db83a2776640] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp, totalSize=35.2 K 2023-04-05 04:58:22,926 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:58:22,926 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33704 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:58:22,926 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=C 2023-04-05 04:58:22,926 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): 3fee67ac8cf8f50b334cb2fd8fafdea1/B is initiating minor compaction (all files) 2023-04-05 04:58:22,926 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:58:22,926 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3fee67ac8cf8f50b334cb2fd8fafdea1/B in TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:22,926 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 460a4fc48c494b2bae3bb5d171b8c7ac, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1680670699924 2023-04-05 04:58:22,926 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/6070a8e7582e4ae49a661844d7e36abb, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/7a0059aa79404f71ae4debec48313c34, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/6162d88224b74d399892061a4df62e49] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp, totalSize=32.9 K 2023-04-05 04:58:22,927 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 687b79f194a84eaca138bb2e41122b0e, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1680670700250 2023-04-05 04:58:22,927 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 6070a8e7582e4ae49a661844d7e36abb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1680670699924 2023-04-05 04:58:22,927 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting e42965417dd5439d88f8db83a2776640, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1680670702073 2023-04-05 04:58:22,927 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 7a0059aa79404f71ae4debec48313c34, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1680670700250 2023-04-05 04:58:22,927 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 6162d88224b74d399892061a4df62e49, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1680670702073 2023-04-05 04:58:22,935 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): 3fee67ac8cf8f50b334cb2fd8fafdea1#B#compaction#393 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:58:22,944 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3fee67ac8cf8f50b334cb2fd8fafdea1#A#compaction#394 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:58:22,957 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=176 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/c6834c986d914be0b736f42ccf9a4c86 2023-04-05 04:58:22,966 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/6c0194446a4243a498995ad40d8d452a as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/6c0194446a4243a498995ad40d8d452a 2023-04-05 04:58:22,969 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3fee67ac8cf8f50b334cb2fd8fafdea1/B of 3fee67ac8cf8f50b334cb2fd8fafdea1 into 6c0194446a4243a498995ad40d8d452a(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:58:22,970 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:22,970 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1., storeName=3fee67ac8cf8f50b334cb2fd8fafdea1/B, priority=13, startTime=1680670702923; duration=0sec 2023-04-05 04:58:22,970 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:58:22,970 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:58:22,970 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33704 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:58:22,971 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): 3fee67ac8cf8f50b334cb2fd8fafdea1/C is initiating minor compaction (all files) 2023-04-05 04:58:22,971 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3fee67ac8cf8f50b334cb2fd8fafdea1/C in TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:22,971 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/9f3f8407116f4ccca92318fe8cd9f321, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/fe29fc5f4094489f9edbba46a5688220, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/c10ea1bc882c4df594285bc681dae059] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp, totalSize=32.9 K 2023-04-05 04:58:22,971 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 9f3f8407116f4ccca92318fe8cd9f321, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1680670699924 2023-04-05 04:58:22,971 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting fe29fc5f4094489f9edbba46a5688220, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1680670700250 2023-04-05 04:58:22,971 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting c10ea1bc882c4df594285bc681dae059, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1680670702073 2023-04-05 04:58:22,975 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): 3fee67ac8cf8f50b334cb2fd8fafdea1#C#compaction#396 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:58:23,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 3fee67ac8cf8f50b334cb2fd8fafdea1 2023-04-05 04:58:23,310 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:23,326 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:23,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59352 deadline: 1680670763322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:23,333 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:23,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59384 deadline: 1680670763326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:23,366 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/7fa596a661be42b68a4581ed27d14d6c as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/7fa596a661be42b68a4581ed27d14d6c 2023-04-05 04:58:23,369 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3fee67ac8cf8f50b334cb2fd8fafdea1/A of 3fee67ac8cf8f50b334cb2fd8fafdea1 into 7fa596a661be42b68a4581ed27d14d6c(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:58:23,369 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:23,369 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1., storeName=3fee67ac8cf8f50b334cb2fd8fafdea1/A, priority=13, startTime=1680670702922; duration=0sec 2023-04-05 04:58:23,369 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:58:23,370 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=176 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/91db3c7f25044c81a871b1bb20f81fb0 2023-04-05 04:58:23,384 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/d5bcda6d8f6847bd826647c3575fd6bc as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/d5bcda6d8f6847bd826647c3575fd6bc 2023-04-05 04:58:23,387 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3fee67ac8cf8f50b334cb2fd8fafdea1/C of 3fee67ac8cf8f50b334cb2fd8fafdea1 into d5bcda6d8f6847bd826647c3575fd6bc(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:58:23,387 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:23,387 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1., storeName=3fee67ac8cf8f50b334cb2fd8fafdea1/C, priority=13, startTime=1680670702923; duration=0sec 2023-04-05 04:58:23,387 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:58:23,433 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:23,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59352 deadline: 1680670763427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:23,439 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:23,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59384 deadline: 1680670763434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:23,637 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:23,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59352 deadline: 1680670763635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:23,642 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:23,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59384 deadline: 1680670763640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:23,778 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=176 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/2b7ca6a890f94e5bb491f34a0b1bd428 2023-04-05 04:58:23,781 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/c6834c986d914be0b736f42ccf9a4c86 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/c6834c986d914be0b736f42ccf9a4c86 2023-04-05 04:58:23,784 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/c6834c986d914be0b736f42ccf9a4c86, entries=150, sequenceid=176, filesize=11.8 K 2023-04-05 04:58:23,785 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/91db3c7f25044c81a871b1bb20f81fb0 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/91db3c7f25044c81a871b1bb20f81fb0 2023-04-05 04:58:23,788 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/91db3c7f25044c81a871b1bb20f81fb0, entries=150, sequenceid=176, filesize=11.8 K 2023-04-05 04:58:23,788 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/2b7ca6a890f94e5bb491f34a0b1bd428 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/2b7ca6a890f94e5bb491f34a0b1bd428 2023-04-05 04:58:23,791 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/2b7ca6a890f94e5bb491f34a0b1bd428, entries=150, sequenceid=176, filesize=11.8 K 2023-04-05 04:58:23,791 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 3fee67ac8cf8f50b334cb2fd8fafdea1 in 865ms, sequenceid=176, compaction requested=false 2023-04-05 04:58:23,791 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:23,791 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:23,791 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:23,792 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:23,792 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:23,792 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:23,792 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:23,792 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:23,793 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:23,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:23,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:23,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:23,793 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:23,793 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:23,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:23,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:23,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:23,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:23,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:23,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:23,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@304ede97[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:23,794 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:23,794 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:23,795 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:23,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:23,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:23,795 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:23,795 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:23,795 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:23,795 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:23,795 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:23,795 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:23,795 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:23,795 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:23,796 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:23,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:23,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:23,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:23,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:23,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:23,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:23,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:23,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:23,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:23,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:23,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:23,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:23,797 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:23,797 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:23,797 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:23,798 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:23,798 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:23,798 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:23,798 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:23,798 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:23,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:23,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:23,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:23,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:23,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:23,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:23,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:23,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:23,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:23,799 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:23,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:23,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:23,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:23,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:23,800 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:23,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:23,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:23,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:23,803 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:23,803 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:23,803 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:23,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:23,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:23,803 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:23,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:23,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:23,803 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:23,803 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:23,803 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:23,803 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:23,803 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:23,803 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:23,803 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:23,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:23,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:23,803 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:23,904 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:23,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:23,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:23,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:23,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:23,906 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:23,906 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:23,906 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:23,906 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:23,906 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:23,907 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:23,907 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:23,907 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:23,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:23,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:23,907 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:23,907 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:23,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:23,908 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:23,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:23,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:23,908 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:23,908 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:23,908 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:23,908 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:23,908 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:23,908 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:23,908 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:23,908 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:23,908 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:23,908 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:23,908 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3fee67ac8cf8f50b334cb2fd8fafdea1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 04:58:23,909 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=A 2023-04-05 04:58:23,909 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:23,909 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=B 2023-04-05 04:58:23,909 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:23,909 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=C 2023-04-05 04:58:23,909 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:23,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 3fee67ac8cf8f50b334cb2fd8fafdea1 2023-04-05 04:58:23,942 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:23,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670703983, max timestamp=1680670703994Num uniques 859; 2023-04-05 04:58:23,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:58:23,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670703983, max timestamp=1680670703994Num uniques 859; 2023-04-05 04:58:23,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670703983, max timestamp=1680670703994Num uniques 859; 2023-04-05 04:58:23,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:58:23,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:58:24,052 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:24,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59384 deadline: 1680670764045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:24,057 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:24,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59352 deadline: 1680670764052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:24,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:24,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59384 deadline: 1680670764153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:24,165 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:24,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59352 deadline: 1680670764158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:24,314 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=189 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/434d5c9fb2514552842ee13f46abe9c3 2023-04-05 04:58:24,359 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:24,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59384 deadline: 1680670764358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:24,371 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:24,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59352 deadline: 1680670764366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:24,400 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:24,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59304 deadline: 1680670764396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:24,402 DEBUG [Thread-2852] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4154 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1., hostname=jenkins-hbase20.apache.org,38091,1680670569206, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 04:58:24,413 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:24,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59330 deadline: 1680670764408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:24,415 DEBUG [Thread-2856] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4162 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1., hostname=jenkins-hbase20.apache.org,38091,1680670569206, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 04:58:24,416 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:24,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59314 deadline: 1680670764412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:24,417 DEBUG [Thread-2858] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4165 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1., hostname=jenkins-hbase20.apache.org,38091,1680670569206, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 04:58:24,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:24,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59384 deadline: 1680670764661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:24,678 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:24,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59352 deadline: 1680670764674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:24,723 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=189 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/c0a4da3b8e58446cba21ca23d8e05cfb 2023-04-05 04:58:25,131 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=189 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/3141b17fbc114ac286a05a232328db98 2023-04-05 04:58:25,134 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/434d5c9fb2514552842ee13f46abe9c3 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/434d5c9fb2514552842ee13f46abe9c3 2023-04-05 04:58:25,137 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/434d5c9fb2514552842ee13f46abe9c3, entries=150, sequenceid=189, filesize=11.8 K 2023-04-05 04:58:25,137 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/c0a4da3b8e58446cba21ca23d8e05cfb as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/c0a4da3b8e58446cba21ca23d8e05cfb 2023-04-05 04:58:25,140 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/c0a4da3b8e58446cba21ca23d8e05cfb, entries=150, sequenceid=189, filesize=11.8 K 2023-04-05 04:58:25,141 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/3141b17fbc114ac286a05a232328db98 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/3141b17fbc114ac286a05a232328db98 2023-04-05 04:58:25,143 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/3141b17fbc114ac286a05a232328db98, entries=150, sequenceid=189, filesize=11.8 K 2023-04-05 04:58:25,144 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 3fee67ac8cf8f50b334cb2fd8fafdea1 in 1236ms, sequenceid=189, compaction requested=true 2023-04-05 04:58:25,144 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:25,144 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:25,144 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:25,144 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:25,144 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,144 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:25,144 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:25,145 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,145 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:25,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:25,145 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,145 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:25,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:25,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:25,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:25,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:25,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ff4a60[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:25,146 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:25,146 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,147 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,147 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:25,147 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:25,147 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:25,147 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,147 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:25,148 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:25,148 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:25,148 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:25,148 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:25,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:25,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:25,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:25,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:25,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:25,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:25,150 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:25,150 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:25,150 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:25,150 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:25,150 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:25,150 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,150 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:25,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:25,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,150 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:25,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:25,151 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:25,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:25,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:25,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,152 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:25,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,153 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,153 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:25,153 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:25,153 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,153 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,153 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,153 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,153 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:25,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:25,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:25,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:25,153 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:25,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:25,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:25,153 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:25,153 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:25,153 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:25,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:25,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:25,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 3fee67ac8cf8f50b334cb2fd8fafdea1 2023-04-05 04:58:25,176 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3fee67ac8cf8f50b334cb2fd8fafdea1 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-05 04:58:25,177 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=A 2023-04-05 04:58:25,177 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-05 04:58:25,177 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=B 2023-04-05 04:58:25,177 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-05 04:58:25,177 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=C 2023-04-05 04:58:25,177 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-05 04:58:25,205 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:25,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59384 deadline: 1680670765196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,214 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:25,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59352 deadline: 1680670765204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,253 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:25,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:25,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:25,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:25,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:25,256 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:25,256 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:25,256 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:25,256 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:25,256 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,256 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:25,257 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,257 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:25,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:25,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:25,257 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,257 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:25,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:25,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:25,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:25,258 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:25,258 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:25,258 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:25,258 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:25,258 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:25,258 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:25,258 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:25,258 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:25,258 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:25,258 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:25,258 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:25,258 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:25,258 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:25,258 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:25,258 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,259 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:25,259 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:25,259 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,259 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:25,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:25,260 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,260 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:25,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:25,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:25,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:25,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:25,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6671a766[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:25,261 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:25,261 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,261 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,261 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:25,261 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:25,261 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,261 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:25,261 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:25,262 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,262 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:25,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:25,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:25,262 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:25,262 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:25,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:25,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:25,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:25,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:25,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:25,264 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:25,264 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:25,264 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:25,264 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:25,264 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:25,264 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,264 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:25,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:25,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,264 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:25,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:25,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:25,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:25,265 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:25,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,266 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:25,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,267 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,267 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:25,267 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:25,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:25,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:25,267 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:25,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:25,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:25,267 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:25,267 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:25,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:25,267 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:25,267 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:25,267 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:25,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:25,267 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,268 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,268 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,308 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:25,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59384 deadline: 1680670765306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,320 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:25,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59352 deadline: 1680670765315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,367 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:25,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:25,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:25,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:25,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:25,370 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:25,370 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:25,370 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:25,370 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:25,370 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,371 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:25,372 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:25,372 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:25,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:25,372 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,372 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:25,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:25,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:25,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:25,373 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:25,373 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:25,373 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:25,373 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:25,373 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:25,373 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:25,373 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:25,373 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:25,373 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:25,374 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:25,374 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:25,374 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:25,374 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:25,374 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:25,374 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,374 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:25,374 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:25,375 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,375 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:25,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:25,375 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,375 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:25,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:25,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:25,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:25,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:25,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30fe87d4[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:25,376 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:25,376 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,376 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,377 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:25,377 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:25,377 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:25,377 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,377 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:25,377 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:25,377 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,377 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:25,377 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:25,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:25,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:25,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:25,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:25,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:25,379 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:25,379 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:25,379 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:25,379 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:25,379 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:25,380 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,380 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,380 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:25,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:25,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:25,380 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:25,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:25,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:25,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:25,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,381 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:25,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,383 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:25,383 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:25,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:25,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:25,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:25,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:25,383 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,383 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:25,383 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:25,383 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:25,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:25,383 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:25,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:25,383 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,383 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:25,383 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,383 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:25,384 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,484 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:25,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:25,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:25,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:25,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:25,486 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:25,486 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:25,486 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:25,486 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:25,486 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,486 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:25,487 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:25,487 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:25,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:25,487 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,487 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:25,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:25,488 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:25,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:25,488 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:25,488 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:25,488 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:25,488 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:25,488 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:25,488 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:25,488 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:25,488 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:25,488 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:25,488 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:25,488 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:25,488 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:25,488 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:25,488 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:25,489 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,489 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:25,489 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:25,489 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,489 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:25,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:25,489 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,489 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:25,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:25,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:25,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:25,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:25,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76825d3a[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:25,491 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:25,491 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,491 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,491 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:25,491 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:25,491 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,491 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:25,491 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:25,492 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:25,492 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:25,492 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,492 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:25,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:25,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:25,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:25,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:25,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:25,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:25,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:25,494 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:25,494 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:25,494 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:25,494 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:25,494 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:25,495 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,495 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:25,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:25,495 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:25,495 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:25,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:25,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:25,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:25,496 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:25,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,498 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,498 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:25,498 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:25,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:25,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:25,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:25,498 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:25,498 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:25,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:25,498 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:25,498 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:25,498 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:25,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:25,498 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,498 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,498 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,498 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:25,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:25,514 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:25,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59384 deadline: 1680670765510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,524 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:25,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59352 deadline: 1680670765522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,583 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=227 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/a5e7f27f76974c489512a9af97626dea 2023-04-05 04:58:25,598 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:25,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:25,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:25,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:25,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:25,600 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:25,600 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:25,600 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:25,600 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:25,601 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,601 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:25,602 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:25,602 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:25,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:25,602 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,602 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:25,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:25,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:25,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:25,603 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:25,603 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:25,603 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:25,603 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:25,603 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:25,603 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:25,604 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:25,604 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:25,604 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:25,604 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:25,604 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:25,604 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:25,604 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:25,604 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:25,604 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,604 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:25,604 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:25,605 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,605 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:25,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:25,605 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,605 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:25,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:25,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:25,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:25,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:25,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dd5a5c0[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:25,606 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:25,606 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,606 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,607 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:25,607 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:25,607 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:25,607 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,607 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:25,607 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,607 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:25,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:25,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:25,607 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:25,608 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:25,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:25,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:25,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:25,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:25,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:25,609 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:25,609 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:25,609 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:25,609 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:25,609 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:25,610 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,610 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:25,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:25,610 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:25,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:25,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:25,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:25,611 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:25,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,612 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:25,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,613 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:25,613 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:25,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:25,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:25,613 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:25,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:25,613 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:25,613 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:25,613 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:25,613 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:25,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:25,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:25,613 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:25,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:25,613 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,613 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,613 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,614 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,713 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:25,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:25,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:25,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:25,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:25,716 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:25,716 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:25,716 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:25,716 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:25,717 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,717 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:25,717 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:25,717 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:25,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:25,718 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,718 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:25,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:25,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:25,718 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:25,718 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:25,719 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:25,719 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:25,719 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:25,719 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:25,719 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:25,719 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:25,719 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:25,719 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:25,719 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:25,719 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:25,719 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:25,719 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:25,719 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,719 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:25,719 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:25,720 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:25,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:25,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:25,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:25,721 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,721 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,721 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:25,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:25,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:25,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6674798b[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:25,721 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:25,722 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,723 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,723 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:25,723 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:25,723 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:25,723 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,723 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:25,723 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,723 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:25,724 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:25,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:25,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:25,724 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:25,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:25,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:25,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:25,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:25,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:25,725 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:25,725 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:25,725 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:25,725 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:25,725 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:25,728 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,728 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,728 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:25,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:25,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:25,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,728 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:25,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:25,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:25,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:25,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,730 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:25,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,733 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:25,733 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:25,733 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:25,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:25,733 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:25,733 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:25,733 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:25,734 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,734 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,734 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,734 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:25,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:25,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:25,734 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:25,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:25,734 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:25,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:25,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:25,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59384 deadline: 1680670765817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,834 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:25,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59352 deadline: 1680670765825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,834 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:25,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:25,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:25,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:25,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:25,837 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:25,837 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:25,837 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:25,837 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:25,838 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,838 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:25,838 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:25,838 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:25,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:25,838 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,839 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:25,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:25,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:25,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:25,839 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:25,839 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:25,839 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:25,839 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:25,839 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:25,839 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:25,840 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:25,840 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:25,840 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:25,840 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:25,840 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:25,840 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:25,840 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:25,840 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:25,840 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,840 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:25,840 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:25,841 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,841 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:25,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:25,841 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,841 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:25,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:25,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:25,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:25,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:25,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@227082b6[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:25,842 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:25,842 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,843 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,843 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:25,843 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:25,843 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:25,843 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,843 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:25,843 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,843 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:25,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:25,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:25,843 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:25,844 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:25,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:25,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:25,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:25,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:25,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:25,845 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:25,845 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:25,845 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:25,845 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:25,845 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:25,848 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,848 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:25,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,848 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:25,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:25,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:25,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:25,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,849 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:25,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:25,850 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:25,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,851 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,851 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:25,851 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:25,852 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:25,852 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:25,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:25,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:25,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:25,852 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,852 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,852 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,852 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:25,852 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:25,852 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:25,852 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:25,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:25,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:25,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:25,952 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:25,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:25,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:25,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:25,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:25,955 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:25,955 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:25,955 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:25,955 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:25,956 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,956 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:25,956 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:25,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:25,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:25,956 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:25,957 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,957 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:25,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:25,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:25,957 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:25,957 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:25,957 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:25,957 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:25,957 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:25,957 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:25,957 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:25,958 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:25,958 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:25,958 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:25,958 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:25,958 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:25,958 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:25,958 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:25,958 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,958 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:25,958 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:25,959 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,959 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:25,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:25,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:25,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:25,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:25,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:25,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46ff604f[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:25,960 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:25,960 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,960 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,960 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:25,960 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,961 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:25,961 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:25,961 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:25,961 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,961 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:25,961 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,961 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:25,961 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:25,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:25,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:25,961 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:25,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:25,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:25,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:25,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:25,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:25,963 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:25,963 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:25,963 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:25,963 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:25,963 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:25,964 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,964 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:25,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:25,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,964 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:25,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:25,965 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:25,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:25,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:25,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,965 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:25,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:25,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,968 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,968 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:25,968 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:25,968 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:25,968 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:25,968 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:25,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:25,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:25,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:25,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:25,968 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:25,968 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:25,968 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:25,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:25,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:25,968 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:25,968 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:25,968 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:25,991 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=227 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/73eaeef6407b47b395ceb3ecabd5b23a 2023-04-05 04:58:26,008 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=227 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/b5e081fd4c504c1994d21018e73b9566 2023-04-05 04:58:26,011 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/a5e7f27f76974c489512a9af97626dea as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/a5e7f27f76974c489512a9af97626dea 2023-04-05 04:58:26,014 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/a5e7f27f76974c489512a9af97626dea, entries=200, sequenceid=227, filesize=14.1 K 2023-04-05 04:58:26,015 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/73eaeef6407b47b395ceb3ecabd5b23a as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/73eaeef6407b47b395ceb3ecabd5b23a 2023-04-05 04:58:26,018 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/73eaeef6407b47b395ceb3ecabd5b23a, entries=150, sequenceid=227, filesize=11.8 K 2023-04-05 04:58:26,019 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/b5e081fd4c504c1994d21018e73b9566 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/b5e081fd4c504c1994d21018e73b9566 2023-04-05 04:58:26,022 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/b5e081fd4c504c1994d21018e73b9566, entries=150, sequenceid=227, filesize=11.8 K 2023-04-05 04:58:26,022 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for 3fee67ac8cf8f50b334cb2fd8fafdea1 in 846ms, sequenceid=227, compaction requested=true 2023-04-05 04:58:26,022 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:26,023 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:58:26,023 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-05 04:58:26,023 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-05 04:58:26,023 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:58:26,023 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:58:26,024 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50876 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:58:26,024 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): 3fee67ac8cf8f50b334cb2fd8fafdea1/A is initiating minor compaction (all files) 2023-04-05 04:58:26,024 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3fee67ac8cf8f50b334cb2fd8fafdea1/A in TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:26,024 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/7fa596a661be42b68a4581ed27d14d6c, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/c6834c986d914be0b736f42ccf9a4c86, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/434d5c9fb2514552842ee13f46abe9c3, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/a5e7f27f76974c489512a9af97626dea] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp, totalSize=49.7 K 2023-04-05 04:58:26,024 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48486 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:58:26,024 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): 3fee67ac8cf8f50b334cb2fd8fafdea1/B is initiating minor compaction (all files) 2023-04-05 04:58:26,024 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 7fa596a661be42b68a4581ed27d14d6c, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1680670702073 2023-04-05 04:58:26,024 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3fee67ac8cf8f50b334cb2fd8fafdea1/B in TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:26,025 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/6c0194446a4243a498995ad40d8d452a, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/91db3c7f25044c81a871b1bb20f81fb0, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/c0a4da3b8e58446cba21ca23d8e05cfb, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/73eaeef6407b47b395ceb3ecabd5b23a] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp, totalSize=47.3 K 2023-04-05 04:58:26,025 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting c6834c986d914be0b736f42ccf9a4c86, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1680670702165 2023-04-05 04:58:26,025 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 6c0194446a4243a498995ad40d8d452a, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1680670702073 2023-04-05 04:58:26,025 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 434d5c9fb2514552842ee13f46abe9c3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1680670703309 2023-04-05 04:58:26,025 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 91db3c7f25044c81a871b1bb20f81fb0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1680670702165 2023-04-05 04:58:26,025 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting a5e7f27f76974c489512a9af97626dea, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1680670704024 2023-04-05 04:58:26,025 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting c0a4da3b8e58446cba21ca23d8e05cfb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1680670703309 2023-04-05 04:58:26,026 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 73eaeef6407b47b395ceb3ecabd5b23a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1680670704024 2023-04-05 04:58:26,033 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3fee67ac8cf8f50b334cb2fd8fafdea1#A#compaction#404 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:58:26,033 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): 3fee67ac8cf8f50b334cb2fd8fafdea1#B#compaction#405 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:58:26,069 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:26,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:26,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:26,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:26,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:26,071 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:26,071 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:26,071 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:26,071 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:26,071 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:26,071 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:26,072 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:26,072 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:26,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:26,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:26,072 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:26,072 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:26,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:26,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:26,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:26,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:26,073 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:26,073 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:26,073 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:26,073 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:26,073 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:26,073 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:26,073 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:26,073 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:26,073 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:26,073 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:26,073 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3fee67ac8cf8f50b334cb2fd8fafdea1 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-05 04:58:26,074 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=A 2023-04-05 04:58:26,074 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:26,074 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=B 2023-04-05 04:58:26,074 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:26,074 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=C 2023-04-05 04:58:26,074 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:26,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 3fee67ac8cf8f50b334cb2fd8fafdea1 2023-04-05 04:58:26,331 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:26,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670706368, max timestamp=1680670706382Num uniques 859; 2023-04-05 04:58:26,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:58:26,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670706368, max timestamp=1680670706382Num uniques 859; 2023-04-05 04:58:26,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:58:26,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670706368, max timestamp=1680670706382Num uniques 859; 2023-04-05 04:58:26,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:58:26,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:26,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59352 deadline: 1680670766430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:26,438 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:26,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59384 deadline: 1680670766431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:26,464 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/291c444351d34c63aa187f0e07f68e74 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/291c444351d34c63aa187f0e07f68e74 2023-04-05 04:58:26,469 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/88e3467cfe374473979b1489d8f8bc4d as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/88e3467cfe374473979b1489d8f8bc4d 2023-04-05 04:58:26,469 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3fee67ac8cf8f50b334cb2fd8fafdea1/A of 3fee67ac8cf8f50b334cb2fd8fafdea1 into 291c444351d34c63aa187f0e07f68e74(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:58:26,469 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:26,469 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1., storeName=3fee67ac8cf8f50b334cb2fd8fafdea1/A, priority=12, startTime=1680670706022; duration=0sec 2023-04-05 04:58:26,469 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:58:26,469 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:58:26,471 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48486 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:58:26,471 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): 3fee67ac8cf8f50b334cb2fd8fafdea1/C is initiating minor compaction (all files) 2023-04-05 04:58:26,471 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3fee67ac8cf8f50b334cb2fd8fafdea1/C in TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:26,471 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/d5bcda6d8f6847bd826647c3575fd6bc, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/2b7ca6a890f94e5bb491f34a0b1bd428, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/3141b17fbc114ac286a05a232328db98, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/b5e081fd4c504c1994d21018e73b9566] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp, totalSize=47.3 K 2023-04-05 04:58:26,471 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting d5bcda6d8f6847bd826647c3575fd6bc, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1680670702073 2023-04-05 04:58:26,472 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 2b7ca6a890f94e5bb491f34a0b1bd428, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1680670702165 2023-04-05 04:58:26,472 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 3141b17fbc114ac286a05a232328db98, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1680670703309 2023-04-05 04:58:26,472 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting b5e081fd4c504c1994d21018e73b9566, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1680670704024 2023-04-05 04:58:26,473 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3fee67ac8cf8f50b334cb2fd8fafdea1/B of 3fee67ac8cf8f50b334cb2fd8fafdea1 into 88e3467cfe374473979b1489d8f8bc4d(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:58:26,473 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:26,473 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1., storeName=3fee67ac8cf8f50b334cb2fd8fafdea1/B, priority=12, startTime=1680670706023; duration=0sec 2023-04-05 04:58:26,473 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:58:26,477 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3fee67ac8cf8f50b334cb2fd8fafdea1#C#compaction#407 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:58:26,480 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=236 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/fb87fedc1c454dc885dd6500bac26807 2023-04-05 04:58:26,488 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/1509c28b0f5c44ddb24f5edcfc438daa as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/1509c28b0f5c44ddb24f5edcfc438daa 2023-04-05 04:58:26,490 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=236 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/56141f5870f24bfc8e3f5316d73eeb82 2023-04-05 04:58:26,493 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3fee67ac8cf8f50b334cb2fd8fafdea1/C of 3fee67ac8cf8f50b334cb2fd8fafdea1 into 1509c28b0f5c44ddb24f5edcfc438daa(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:58:26,493 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:26,493 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1., storeName=3fee67ac8cf8f50b334cb2fd8fafdea1/C, priority=12, startTime=1680670706023; duration=0sec 2023-04-05 04:58:26,493 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:58:26,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:26,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59352 deadline: 1680670766537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:26,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:26,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59384 deadline: 1680670766539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:26,744 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:26,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59384 deadline: 1680670766742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:26,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:26,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59352 deadline: 1680670766743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:26,900 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=236 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/3f6fcdb60f154bb0af6803024a2f1caf 2023-04-05 04:58:26,903 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/fb87fedc1c454dc885dd6500bac26807 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/fb87fedc1c454dc885dd6500bac26807 2023-04-05 04:58:26,906 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/fb87fedc1c454dc885dd6500bac26807, entries=150, sequenceid=236, filesize=11.8 K 2023-04-05 04:58:26,907 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/56141f5870f24bfc8e3f5316d73eeb82 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/56141f5870f24bfc8e3f5316d73eeb82 2023-04-05 04:58:26,910 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/56141f5870f24bfc8e3f5316d73eeb82, entries=150, sequenceid=236, filesize=11.8 K 2023-04-05 04:58:26,911 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/3f6fcdb60f154bb0af6803024a2f1caf as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/3f6fcdb60f154bb0af6803024a2f1caf 2023-04-05 04:58:26,913 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/3f6fcdb60f154bb0af6803024a2f1caf, entries=150, sequenceid=236, filesize=11.8 K 2023-04-05 04:58:26,914 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 3fee67ac8cf8f50b334cb2fd8fafdea1 in 841ms, sequenceid=236, compaction requested=false 2023-04-05 04:58:26,914 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:26,914 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:26,914 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:26,914 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:26,914 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:26,914 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:26,914 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:26,915 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:26,915 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:26,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:26,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:26,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:26,915 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:26,915 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:26,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:26,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:26,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:26,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:26,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:26,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:26,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66a426f5[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:26,916 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:26,917 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:26,917 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:26,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:26,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:26,917 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:26,917 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:26,917 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:26,917 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:26,917 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:26,919 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:26,919 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:26,919 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:26,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:26,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:26,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:26,919 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:26,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:26,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:26,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:26,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:26,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:26,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:26,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:26,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:26,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:26,921 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:26,921 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:26,921 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:26,921 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:26,921 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:26,922 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:26,922 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:26,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:26,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:26,922 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:26,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:26,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:26,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:26,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:26,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:26,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:26,922 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:26,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:26,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:26,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:26,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:26,923 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:26,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:26,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:26,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:26,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:26,924 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:26,924 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:26,925 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:26,925 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:26,925 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:26,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:26,925 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:26,925 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:26,925 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:26,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:26,925 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:26,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:26,925 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:26,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:26,925 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:26,925 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:26,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:26,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:26,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:27,025 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:27,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:27,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:27,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:27,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:27,027 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:27,028 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:27,028 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:27,028 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:27,028 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:27,028 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:27,029 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:27,029 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:27,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:27,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:27,029 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:27,029 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:27,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:27,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:27,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:27,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:27,029 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:27,030 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:27,030 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:27,030 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:27,030 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:27,030 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:27,030 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:27,030 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:27,030 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:27,030 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:27,030 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3fee67ac8cf8f50b334cb2fd8fafdea1 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-05 04:58:27,030 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=A 2023-04-05 04:58:27,030 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:58:27,030 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=B 2023-04-05 04:58:27,031 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:58:27,031 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=C 2023-04-05 04:58:27,031 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:58:27,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 3fee67ac8cf8f50b334cb2fd8fafdea1 2023-04-05 04:58:27,050 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:27,064 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:27,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59352 deadline: 1680670767062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:27,067 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:27,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59384 deadline: 1680670767063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:27,165 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:27,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59352 deadline: 1680670767165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:27,171 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:27,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59384 deadline: 1680670767168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:27,372 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:27,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59352 deadline: 1680670767367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:27,377 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:27,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59384 deadline: 1680670767373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:27,438 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=276 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/fca4f5cacecd4e2681a2fddff82cea1b 2023-04-05 04:58:27,674 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:27,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59352 deadline: 1680670767674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:27,680 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:27,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59384 deadline: 1680670767678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:27,846 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=276 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/470ea5f09cc54b31b64f11cf445c5430 2023-04-05 04:58:28,181 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:28,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59352 deadline: 1680670768177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:28,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:28,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59384 deadline: 1680670768180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:28,255 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=276 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/14e6f29ddaed4378900dd3afd76b35ad 2023-04-05 04:58:28,259 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/fca4f5cacecd4e2681a2fddff82cea1b as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/fca4f5cacecd4e2681a2fddff82cea1b 2023-04-05 04:58:28,261 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/fca4f5cacecd4e2681a2fddff82cea1b, entries=150, sequenceid=276, filesize=11.9 K 2023-04-05 04:58:28,262 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/470ea5f09cc54b31b64f11cf445c5430 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/470ea5f09cc54b31b64f11cf445c5430 2023-04-05 04:58:28,264 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/470ea5f09cc54b31b64f11cf445c5430, entries=150, sequenceid=276, filesize=11.9 K 2023-04-05 04:58:28,265 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/14e6f29ddaed4378900dd3afd76b35ad as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/14e6f29ddaed4378900dd3afd76b35ad 2023-04-05 04:58:28,267 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/14e6f29ddaed4378900dd3afd76b35ad, entries=150, sequenceid=276, filesize=11.9 K 2023-04-05 04:58:28,268 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=46.96 KB/48090 for 3fee67ac8cf8f50b334cb2fd8fafdea1 in 1238ms, sequenceid=276, compaction requested=true 2023-04-05 04:58:28,268 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:28,268 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:28,268 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:28,268 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:28,268 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:28,268 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:28,268 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:28,269 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:28,269 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:28,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:28,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:28,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:28,269 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:28,270 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:28,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:28,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:28,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:28,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:28,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:28,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:28,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2359b38b[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:28,272 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:28,272 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:28,272 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:28,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:28,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:28,272 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:28,272 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:28,272 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:28,272 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:28,272 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:28,273 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:28,273 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:28,273 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:28,273 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:28,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:28,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:28,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:28,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:28,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:28,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:28,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:28,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:28,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:28,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:28,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:28,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:28,275 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:28,275 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:28,275 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:28,275 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:28,275 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:28,276 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:28,276 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:28,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:28,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:28,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:28,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:28,276 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:28,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:28,276 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:28,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:28,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:28,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:28,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:28,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:28,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:28,277 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:28,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:28,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:28,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:28,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:28,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:28,278 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:28,278 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:28,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:28,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:28,278 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:28,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:28,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:28,278 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:28,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:28,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:28,278 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:28,279 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:28,279 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:28,279 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:28,279 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:28,279 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:28,279 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:28,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:28,279 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:28,379 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:28,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:28,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:28,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:28,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:28,382 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:28,382 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:28,382 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:28,382 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:28,382 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:28,382 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:28,383 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:28,383 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:28,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:28,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:28,383 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:28,383 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:28,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:28,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:28,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:28,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:28,384 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:28,384 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:28,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:28,384 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:28,384 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:28,384 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:28,384 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:28,384 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:28,385 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:28,385 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:28,385 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3fee67ac8cf8f50b334cb2fd8fafdea1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 04:58:28,385 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=A 2023-04-05 04:58:28,385 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:28,385 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=B 2023-04-05 04:58:28,385 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:28,385 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=C 2023-04-05 04:58:28,385 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:28,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 3fee67ac8cf8f50b334cb2fd8fafdea1 2023-04-05 04:58:28,420 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:28,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670708450, max timestamp=1680670708467Num uniques 859; 2023-04-05 04:58:28,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:58:28,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670708450, max timestamp=1680670708467Num uniques 859; 2023-04-05 04:58:28,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:58:28,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670708450, max timestamp=1680670708467Num uniques 859; 2023-04-05 04:58:28,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:58:28,508 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:28,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59304 deadline: 1680670768498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:28,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:28,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59330 deadline: 1680670768503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:28,515 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:28,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59314 deadline: 1680670768509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:28,611 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:28,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59304 deadline: 1680670768609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:28,618 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:28,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59330 deadline: 1680670768615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:28,619 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:28,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59314 deadline: 1680670768616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:28,792 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/e230cebb6dc640a6b862761e38bf52ca 2023-04-05 04:58:28,818 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:28,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59304 deadline: 1680670768813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:28,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:28,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59330 deadline: 1680670768820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:28,826 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:28,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59314 deadline: 1680670768820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:29,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59304 deadline: 1680670769119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,131 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:29,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59314 deadline: 1680670769126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,133 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:29,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59330 deadline: 1680670769127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,187 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:29,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59352 deadline: 1680670769182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,190 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:29,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59384 deadline: 1680670769185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,200 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/dd546e465a32415bbdf2bce585d6e4d0 2023-04-05 04:58:29,608 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/99b8111047b94695b962ef2103f3f5aa 2023-04-05 04:58:29,611 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/e230cebb6dc640a6b862761e38bf52ca as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/e230cebb6dc640a6b862761e38bf52ca 2023-04-05 04:58:29,614 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/e230cebb6dc640a6b862761e38bf52ca, entries=150, sequenceid=286, filesize=11.9 K 2023-04-05 04:58:29,615 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/dd546e465a32415bbdf2bce585d6e4d0 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/dd546e465a32415bbdf2bce585d6e4d0 2023-04-05 04:58:29,617 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/dd546e465a32415bbdf2bce585d6e4d0, entries=150, sequenceid=286, filesize=11.9 K 2023-04-05 04:58:29,618 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/99b8111047b94695b962ef2103f3f5aa as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/99b8111047b94695b962ef2103f3f5aa 2023-04-05 04:58:29,623 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/99b8111047b94695b962ef2103f3f5aa, entries=150, sequenceid=286, filesize=11.9 K 2023-04-05 04:58:29,623 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 3fee67ac8cf8f50b334cb2fd8fafdea1 in 1238ms, sequenceid=286, compaction requested=true 2023-04-05 04:58:29,623 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:29,623 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:29,623 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:29,623 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:29,624 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,624 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:29,624 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:29,624 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:29,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:29,624 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:29,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:29,625 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:29,625 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:29,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:29,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:29,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:29,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:29,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59ae064[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:29,626 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:29,626 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:29,626 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:29,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:29,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:29,627 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:29,627 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:29,627 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:29,627 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,627 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:29,627 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,627 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:29,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:29,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:29,628 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:29,628 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:29,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:29,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:29,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:29,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:29,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:29,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:29,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:29,630 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:29,630 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:29,630 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:29,630 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:29,630 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:29,630 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:29,630 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:29,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:29,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:29,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:29,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:29,631 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:29,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:29,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:29,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:29,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:29,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:29,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:29,631 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:29,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:29,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,632 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:29,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:29,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,634 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:29,634 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:29,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:29,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:29,634 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:29,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:29,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:29,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:29,634 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:29,634 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:29,634 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:29,634 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,634 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:29,634 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:29,634 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:29,634 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:29,634 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:29,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:29,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 3fee67ac8cf8f50b334cb2fd8fafdea1 2023-04-05 04:58:29,636 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3fee67ac8cf8f50b334cb2fd8fafdea1 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-05 04:58:29,636 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=A 2023-04-05 04:58:29,636 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:58:29,636 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=B 2023-04-05 04:58:29,636 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:58:29,636 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=C 2023-04-05 04:58:29,636 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:58:29,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:29,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59330 deadline: 1680670769667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,677 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:29,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59304 deadline: 1680670769668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,677 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:29,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59314 deadline: 1680670769668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,735 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:29,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:29,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:29,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:29,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:29,737 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:29,737 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:29,737 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:29,737 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:29,738 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:29,738 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:29,742 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,742 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:29,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:29,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:29,743 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,743 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:29,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:29,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:29,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:29,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:29,743 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:29,743 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:29,744 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:29,744 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:29,744 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:29,744 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:29,744 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:29,744 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:29,744 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:29,744 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:29,744 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:29,744 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:29,744 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:29,744 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,744 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:29,744 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:29,745 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:29,745 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:29,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:29,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:29,746 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:29,746 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:29,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:29,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:29,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:29,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:29,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@385a1580[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:29,747 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:29,747 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:29,747 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:29,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:29,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:29,747 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:29,747 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:29,747 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,747 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:29,747 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:29,748 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,748 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:29,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:29,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:29,748 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:29,748 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:29,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:29,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:29,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:29,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:29,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:29,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:29,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:29,750 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:29,750 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:29,750 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:29,750 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:29,750 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:29,751 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:29,751 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:29,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:29,751 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:29,751 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:29,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:29,752 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:29,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:29,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:29,754 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,754 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:29,754 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:29,754 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:29,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:29,754 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:29,754 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:29,754 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:29,754 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:29,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:29,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:29,754 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:29,754 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:29,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:29,754 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:29,755 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:29,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:29,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:29,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:29,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59330 deadline: 1680670769776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,785 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:29,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59304 deadline: 1680670769778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,786 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:29,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59314 deadline: 1680670769778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,855 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:29,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:29,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:29,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:29,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:29,857 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:29,857 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:29,857 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:29,857 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:29,858 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:29,858 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:29,859 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,859 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:29,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:29,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:29,859 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,859 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:29,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:29,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:29,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:29,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:29,859 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:29,859 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:29,860 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:29,860 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:29,860 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:29,860 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:29,860 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:29,860 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:29,860 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:29,860 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:29,860 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:29,860 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:29,860 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:29,860 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:29,860 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,860 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:29,860 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:29,861 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,861 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:29,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:29,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:29,861 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:29,861 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:29,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:29,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:29,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:29,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:29,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:29,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f8e1ff0[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:29,862 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:29,863 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:29,863 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:29,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:29,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:29,863 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:29,863 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:29,863 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:29,863 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,864 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:29,864 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,864 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:29,864 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:29,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:29,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:29,864 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:29,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:29,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:29,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:29,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:29,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:29,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:29,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:29,866 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:29,866 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:29,866 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:29,866 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:29,866 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:29,867 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:29,867 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:29,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:29,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:29,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:29,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:29,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:29,867 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:29,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:29,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:29,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:29,867 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:29,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:29,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:29,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:29,868 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:29,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:29,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,869 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:29,869 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:29,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:29,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:29,869 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:29,869 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:29,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:29,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:29,869 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:29,869 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:29,869 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:29,869 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:29,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:29,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:29,869 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,870 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:29,870 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:29,870 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:29,970 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:29,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:29,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:29,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:29,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:29,973 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:29,973 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:29,973 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:29,973 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:29,973 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:29,973 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:29,974 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:29,974 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:29,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:29,974 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,974 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:29,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:29,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:29,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:29,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:29,975 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:29,975 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:29,975 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:29,975 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:29,975 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:29,975 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:29,975 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:29,975 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:29,975 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:29,975 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:29,975 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:29,975 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:29,975 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:29,975 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:29,975 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,975 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:29,975 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:29,976 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,976 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:29,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:29,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:29,976 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:29,976 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:29,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:29,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:29,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:29,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:29,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:29,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11a52e61[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:29,977 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:29,977 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:29,978 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:29,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:29,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:29,978 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,978 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:29,978 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:29,978 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:29,978 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:29,979 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,979 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:29,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:29,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:29,979 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:29,979 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:29,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:29,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:29,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:29,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:29,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:29,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:29,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:29,981 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:29,981 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:29,981 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:29,981 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:29,981 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:29,982 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:29,982 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:29,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:29,982 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:29,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:29,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:29,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:29,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:29,982 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:29,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:29,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:29,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:29,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:29,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:29,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:29,983 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:29,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:29,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,987 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:29,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59330 deadline: 1680670769985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,991 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:29,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59304 deadline: 1680670769986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,991 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:29,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59314 deadline: 1680670769986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,996 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:29,996 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:29,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:29,996 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:29,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:29,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:29,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:29,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:29,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:29,996 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:29,996 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:29,996 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:29,997 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:29,996 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:29,997 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:29,997 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:29,997 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:29,997 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:30,064 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=323 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/a62e2a52afa243d38b79ffee41e1f568 2023-04-05 04:58:30,097 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:30,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:30,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:30,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:30,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:30,099 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:30,100 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:30,100 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:30,100 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:30,100 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:30,100 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:30,101 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:30,101 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:30,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:30,101 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,101 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:30,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:30,101 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:30,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:30,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:30,101 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:30,102 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:30,102 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:30,102 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:30,102 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:30,102 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:30,102 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:30,102 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:30,102 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:30,102 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:30,102 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:30,102 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:30,102 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:30,102 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:30,102 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,102 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:30,102 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:30,103 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,103 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:30,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:30,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:30,103 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:30,103 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:30,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:30,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:30,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:30,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:30,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:30,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@436e75f1[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:30,104 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:30,104 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:30,105 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:30,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:30,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:30,105 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:30,105 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:30,105 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:30,105 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,105 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:30,106 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,106 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:30,106 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:30,106 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:30,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:30,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:30,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:30,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:30,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:30,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:30,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:30,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:30,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:30,108 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:30,108 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:30,108 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:30,108 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:30,108 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:30,109 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:30,109 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:30,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:30,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:30,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:30,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:30,109 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:30,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:30,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:30,110 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:30,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:30,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:30,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:30,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:30,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:30,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,110 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:30,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:30,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,112 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,112 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:30,112 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:30,112 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:30,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:30,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:30,112 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,112 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:30,113 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:30,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:30,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:30,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:30,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:30,112 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:30,113 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:30,113 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:30,113 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:30,113 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:30,213 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:30,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:30,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:30,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:30,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:30,216 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:30,216 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:30,216 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:30,216 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:30,216 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:30,216 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:30,217 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,217 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:30,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:30,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:30,217 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,217 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:30,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:30,218 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:30,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:30,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:30,218 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:30,218 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:30,218 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:30,218 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:30,218 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:30,218 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:30,219 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:30,219 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:30,219 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:30,219 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:30,219 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:30,219 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:30,219 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:30,219 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:30,219 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,219 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:30,219 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:30,220 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,220 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:30,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:30,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:30,220 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:30,220 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:30,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:30,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:30,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:30,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:30,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:30,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bf09e09[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:30,222 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:30,222 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:30,222 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:30,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:30,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:30,223 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:30,223 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:30,223 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:30,223 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,223 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:30,224 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:30,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:30,224 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:30,224 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:30,224 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:30,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:30,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:30,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:30,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:30,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:30,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:30,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:30,225 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:30,225 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:30,225 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:30,225 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:30,225 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:30,226 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:30,226 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:30,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:30,226 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:30,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:30,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:30,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:30,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:30,227 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:30,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:30,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:30,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:30,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:30,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:30,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:30,228 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:30,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:30,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,229 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:30,229 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:30,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:30,229 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:30,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:30,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:30,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:30,229 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:30,229 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:30,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:30,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:30,229 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:30,229 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:30,229 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,230 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:30,230 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:30,230 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:30,230 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:30,291 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:30,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59330 deadline: 1680670770289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,298 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:30,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59304 deadline: 1680670770293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,298 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:30,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59314 deadline: 1680670770293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,330 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:30,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:30,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:30,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:30,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:30,332 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:30,332 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:30,332 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:30,332 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:30,333 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:30,333 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:30,333 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,333 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:30,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:30,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:30,333 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,333 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:30,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:30,334 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:30,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:30,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:30,334 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:30,334 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:30,334 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:30,334 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:30,334 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:30,335 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:30,335 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:30,335 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:30,335 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:30,335 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:30,335 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:30,335 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:30,335 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:30,335 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:30,335 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,335 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:30,335 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:30,336 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:30,336 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:30,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:30,336 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:30,336 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:30,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:30,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:30,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:30,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:30,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:30,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@240e7518[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:30,337 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:30,337 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:30,338 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:30,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:30,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:30,338 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:30,338 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:30,338 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:30,338 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,338 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:30,338 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,338 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:30,338 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:30,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:30,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:30,339 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:30,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:30,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:30,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:30,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:30,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:30,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:30,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:30,340 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:30,340 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:30,340 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:30,340 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:30,340 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:30,341 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:30,341 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:30,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:30,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:30,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:30,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:30,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:30,341 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:30,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:30,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:30,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:30,341 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:30,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:30,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:30,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:30,342 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:30,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:30,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,344 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:30,344 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,344 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:30,344 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:30,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:30,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:30,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:30,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:30,344 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:30,344 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:30,344 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:30,344 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,344 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:30,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:30,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:30,344 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:30,344 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:30,344 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:30,444 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:30,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:30,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:30,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:30,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:30,447 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:30,447 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:30,447 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:30,447 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:30,448 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:30,448 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:30,448 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,448 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:30,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:30,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:30,448 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,448 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:30,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:30,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:30,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:30,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:30,449 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:30,449 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:30,449 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:30,449 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:30,449 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:30,449 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:30,449 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:30,449 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:30,449 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:30,449 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:30,450 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:30,450 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:30,450 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:30,450 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:30,450 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,450 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:30,450 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:30,450 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,450 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:30,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:30,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:30,451 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:30,451 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:30,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:30,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:30,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:30,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:30,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:30,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ce091de[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:30,452 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:30,452 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:30,453 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:30,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:30,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:30,453 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:30,453 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:30,453 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:30,453 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,453 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:30,454 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:30,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:30,454 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:30,454 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:30,454 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:30,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:30,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:30,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:30,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:30,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:30,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:30,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:30,455 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:30,455 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:30,455 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:30,456 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:30,456 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:30,456 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:30,456 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:30,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:30,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:30,456 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:30,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:30,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:30,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:30,457 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:30,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:30,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:30,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:30,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:30,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:30,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:30,457 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:30,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:30,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,459 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,459 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:30,459 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:30,459 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,459 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:30,459 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:30,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:30,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:30,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:30,459 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:30,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:30,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:30,459 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:30,459 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:30,459 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:30,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:30,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:30,459 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:30,459 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:30,474 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=323 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/929cabb18e3d48eeaeaa8389d4c6d2d4 2023-04-05 04:58:30,484 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=323 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/33518fc71ce247bbadc780056ca3cede 2023-04-05 04:58:30,486 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/a62e2a52afa243d38b79ffee41e1f568 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/a62e2a52afa243d38b79ffee41e1f568 2023-04-05 04:58:30,489 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/a62e2a52afa243d38b79ffee41e1f568, entries=200, sequenceid=323, filesize=14.3 K 2023-04-05 04:58:30,489 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/929cabb18e3d48eeaeaa8389d4c6d2d4 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/929cabb18e3d48eeaeaa8389d4c6d2d4 2023-04-05 04:58:30,493 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/929cabb18e3d48eeaeaa8389d4c6d2d4, entries=150, sequenceid=323, filesize=11.9 K 2023-04-05 04:58:30,494 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/33518fc71ce247bbadc780056ca3cede as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/33518fc71ce247bbadc780056ca3cede 2023-04-05 04:58:30,496 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/33518fc71ce247bbadc780056ca3cede, entries=150, sequenceid=323, filesize=11.9 K 2023-04-05 04:58:30,497 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=46.96 KB/48090 for 3fee67ac8cf8f50b334cb2fd8fafdea1 in 861ms, sequenceid=323, compaction requested=true 2023-04-05 04:58:30,497 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:30,497 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:58:30,497 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 04:58:30,497 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 04:58:30,497 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:58:30,497 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:58:30,498 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 63548 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 04:58:30,498 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): 3fee67ac8cf8f50b334cb2fd8fafdea1/A is initiating minor compaction (all files) 2023-04-05 04:58:30,498 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3fee67ac8cf8f50b334cb2fd8fafdea1/A in TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:30,498 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/291c444351d34c63aa187f0e07f68e74, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/fb87fedc1c454dc885dd6500bac26807, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/fca4f5cacecd4e2681a2fddff82cea1b, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/e230cebb6dc640a6b862761e38bf52ca, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/a62e2a52afa243d38b79ffee41e1f568] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp, totalSize=62.1 K 2023-04-05 04:58:30,498 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61108 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 04:58:30,498 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): 3fee67ac8cf8f50b334cb2fd8fafdea1/B is initiating minor compaction (all files) 2023-04-05 04:58:30,498 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3fee67ac8cf8f50b334cb2fd8fafdea1/B in TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:30,498 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/88e3467cfe374473979b1489d8f8bc4d, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/56141f5870f24bfc8e3f5316d73eeb82, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/470ea5f09cc54b31b64f11cf445c5430, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/dd546e465a32415bbdf2bce585d6e4d0, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/929cabb18e3d48eeaeaa8389d4c6d2d4] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp, totalSize=59.7 K 2023-04-05 04:58:30,499 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 291c444351d34c63aa187f0e07f68e74, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1680670704024 2023-04-05 04:58:30,499 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 88e3467cfe374473979b1489d8f8bc4d, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1680670704024 2023-04-05 04:58:30,499 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting fb87fedc1c454dc885dd6500bac26807, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1680670705183 2023-04-05 04:58:30,499 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 56141f5870f24bfc8e3f5316d73eeb82, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1680670705183 2023-04-05 04:58:30,499 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting fca4f5cacecd4e2681a2fddff82cea1b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1680670706421 2023-04-05 04:58:30,499 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 470ea5f09cc54b31b64f11cf445c5430, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1680670706421 2023-04-05 04:58:30,499 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting e230cebb6dc640a6b862761e38bf52ca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1680670707054 2023-04-05 04:58:30,499 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting dd546e465a32415bbdf2bce585d6e4d0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1680670707054 2023-04-05 04:58:30,500 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting a62e2a52afa243d38b79ffee41e1f568, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=323, earliestPutTs=1680670708496 2023-04-05 04:58:30,500 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 929cabb18e3d48eeaeaa8389d4c6d2d4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=323, earliestPutTs=1680670708501 2023-04-05 04:58:30,507 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): 3fee67ac8cf8f50b334cb2fd8fafdea1#B#compaction#420 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:58:30,507 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3fee67ac8cf8f50b334cb2fd8fafdea1#A#compaction#419 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:58:30,513 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/4660535f1858438eaffd68513d7c531b as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/4660535f1858438eaffd68513d7c531b 2023-04-05 04:58:30,516 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 3fee67ac8cf8f50b334cb2fd8fafdea1/A of 3fee67ac8cf8f50b334cb2fd8fafdea1 into 4660535f1858438eaffd68513d7c531b(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:58:30,516 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:30,516 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1., storeName=3fee67ac8cf8f50b334cb2fd8fafdea1/A, priority=11, startTime=1680670710497; duration=0sec 2023-04-05 04:58:30,516 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:58:30,516 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 04:58:30,518 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61108 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 04:58:30,518 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): 3fee67ac8cf8f50b334cb2fd8fafdea1/C is initiating minor compaction (all files) 2023-04-05 04:58:30,518 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3fee67ac8cf8f50b334cb2fd8fafdea1/C in TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:30,518 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/1509c28b0f5c44ddb24f5edcfc438daa, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/3f6fcdb60f154bb0af6803024a2f1caf, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/14e6f29ddaed4378900dd3afd76b35ad, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/99b8111047b94695b962ef2103f3f5aa, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/33518fc71ce247bbadc780056ca3cede] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp, totalSize=59.7 K 2023-04-05 04:58:30,518 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 1509c28b0f5c44ddb24f5edcfc438daa, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1680670704024 2023-04-05 04:58:30,518 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 3f6fcdb60f154bb0af6803024a2f1caf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1680670705183 2023-04-05 04:58:30,519 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 14e6f29ddaed4378900dd3afd76b35ad, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1680670706421 2023-04-05 04:58:30,519 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 99b8111047b94695b962ef2103f3f5aa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1680670707054 2023-04-05 04:58:30,519 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 33518fc71ce247bbadc780056ca3cede, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=323, earliestPutTs=1680670708501 2023-04-05 04:58:30,524 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3fee67ac8cf8f50b334cb2fd8fafdea1#C#compaction#421 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:58:30,559 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:30,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:30,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:30,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:30,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:30,562 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:30,562 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:30,562 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:30,562 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:30,563 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:30,563 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:30,564 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,564 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:30,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:30,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:30,564 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,564 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:30,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:30,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:30,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:30,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:30,565 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:30,565 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:30,565 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:30,565 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:30,565 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:30,565 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:30,565 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:30,565 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:30,565 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:30,565 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:30,565 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3fee67ac8cf8f50b334cb2fd8fafdea1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 04:58:30,566 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=A 2023-04-05 04:58:30,566 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:30,566 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=B 2023-04-05 04:58:30,566 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:30,566 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=C 2023-04-05 04:58:30,566 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:30,571 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=334 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/6f316daa848c42069d26a2a7a79a231f 2023-04-05 04:58:30,578 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=334 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/6a0158f455ab42749784822157f88da2 2023-04-05 04:58:30,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 3fee67ac8cf8f50b334cb2fd8fafdea1 2023-04-05 04:58:30,798 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:30,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680670710815, max timestamp=1680670710828Num uniques 859; 2023-04-05 04:58:30,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680670710815, max timestamp=1680670710828Num uniques 859; 2023-04-05 04:58:30,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 04:58:30,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 04:58:30,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680670710815, max timestamp=1680670710828Num uniques 859; 2023-04-05 04:58:30,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-05 04:58:30,861 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:30,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59314 deadline: 1680670770854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,861 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:30,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59304 deadline: 1680670770856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,864 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:30,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59330 deadline: 1680670770861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,913 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/4a8a3a9eb60b42a789ab4b1774340ce5 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/4a8a3a9eb60b42a789ab4b1774340ce5 2023-04-05 04:58:30,917 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 3fee67ac8cf8f50b334cb2fd8fafdea1/B of 3fee67ac8cf8f50b334cb2fd8fafdea1 into 4a8a3a9eb60b42a789ab4b1774340ce5(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:58:30,917 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:30,917 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1., storeName=3fee67ac8cf8f50b334cb2fd8fafdea1/B, priority=11, startTime=1680670710497; duration=0sec 2023-04-05 04:58:30,917 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:58:30,930 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/0e5e4fac52a0495aae6ebef8082b86ed as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/0e5e4fac52a0495aae6ebef8082b86ed 2023-04-05 04:58:30,934 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 3fee67ac8cf8f50b334cb2fd8fafdea1/C of 3fee67ac8cf8f50b334cb2fd8fafdea1 into 0e5e4fac52a0495aae6ebef8082b86ed(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:58:30,934 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:30,934 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1., storeName=3fee67ac8cf8f50b334cb2fd8fafdea1/C, priority=11, startTime=1680670710497; duration=0sec 2023-04-05 04:58:30,934 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:58:30,965 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:30,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59314 deadline: 1680670770962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,965 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:30,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59304 deadline: 1680670770962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,967 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:30,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59330 deadline: 1680670770964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,984 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=334 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/af5b1a9e53934d2fba504b451e699e77 2023-04-05 04:58:30,988 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/6f316daa848c42069d26a2a7a79a231f as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/6f316daa848c42069d26a2a7a79a231f 2023-04-05 04:58:30,991 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/6f316daa848c42069d26a2a7a79a231f, entries=150, sequenceid=334, filesize=11.9 K 2023-04-05 04:58:30,991 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/6a0158f455ab42749784822157f88da2 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/6a0158f455ab42749784822157f88da2 2023-04-05 04:58:30,994 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/6a0158f455ab42749784822157f88da2, entries=150, sequenceid=334, filesize=11.9 K 2023-04-05 04:58:30,995 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/af5b1a9e53934d2fba504b451e699e77 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/af5b1a9e53934d2fba504b451e699e77 2023-04-05 04:58:30,997 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/af5b1a9e53934d2fba504b451e699e77, entries=150, sequenceid=334, filesize=11.9 K 2023-04-05 04:58:30,998 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 3fee67ac8cf8f50b334cb2fd8fafdea1 in 433ms, sequenceid=334, compaction requested=false 2023-04-05 04:58:30,998 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:30,998 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:30,998 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:30,998 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:30,998 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:30,998 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:30,998 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:30,999 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:30,999 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:30,999 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:31,000 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:31,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:31,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:31,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:31,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:31,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:31,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:31,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c11be4e[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:31,001 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:31,001 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:31,002 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:31,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:31,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:31,002 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:31,002 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:31,002 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:31,003 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:31,003 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,003 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:31,003 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:31,003 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:31,003 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:31,003 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:31,003 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:31,003 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:31,003 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:31,003 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:31,004 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:31,004 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:31,004 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:31,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:31,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:31,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:31,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:31,005 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:31,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:31,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:31,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:31,006 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:31,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:31,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,007 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:31,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:31,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:31,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:31,007 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:31,007 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:31,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:31,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:31,007 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,007 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:31,008 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:31,008 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:31,008 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:31,008 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,008 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:31,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:31,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:31,008 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:31,008 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:31,108 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:31,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:31,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:31,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:31,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:31,110 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:31,110 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:31,110 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:31,110 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:31,111 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:31,111 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:31,111 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:31,111 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:31,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:31,111 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,111 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:31,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:31,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:31,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:31,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:31,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:31,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:31,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:31,112 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:31,112 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:31,112 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:31,113 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:31,113 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:31,113 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:31,113 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:31,113 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3fee67ac8cf8f50b334cb2fd8fafdea1 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-05 04:58:31,113 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=A 2023-04-05 04:58:31,113 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:58:31,113 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=B 2023-04-05 04:58:31,113 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:58:31,113 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=C 2023-04-05 04:58:31,113 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:58:31,119 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=371 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/9447931e9c4d4936946ff6e546b01a13 2023-04-05 04:58:31,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 3fee67ac8cf8f50b334cb2fd8fafdea1 2023-04-05 04:58:31,168 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:31,184 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:31,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59304 deadline: 1680670771175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,184 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:31,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59314 deadline: 1680670771178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:31,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59330 deadline: 1680670771179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,200 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:31,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59384 deadline: 1680670771196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,203 DEBUG [Thread-2860] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4138 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1., hostname=jenkins-hbase20.apache.org,38091,1680670569206, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 04:58:31,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:31,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59352 deadline: 1680670771198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,208 DEBUG [Thread-2854] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4144 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1., hostname=jenkins-hbase20.apache.org,38091,1680670569206, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 04:58:31,289 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:31,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59314 deadline: 1680670771285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,289 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:31,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59304 deadline: 1680670771285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,289 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:31,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59330 deadline: 1680670771285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,493 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:31,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59304 deadline: 1680670771490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,493 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:31,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59314 deadline: 1680670771490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,495 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:31,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59330 deadline: 1680670771491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,529 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=371 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/489a8793ff964358927908c277f837dd 2023-04-05 04:58:31,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:31,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59314 deadline: 1680670771794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,800 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:31,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59304 deadline: 1680670771795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,803 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:31,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59330 deadline: 1680670771797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,939 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=371 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/e8acabe08ee94a418d80d3d61ae7824d 2023-04-05 04:58:31,943 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/9447931e9c4d4936946ff6e546b01a13 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/9447931e9c4d4936946ff6e546b01a13 2023-04-05 04:58:31,946 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/9447931e9c4d4936946ff6e546b01a13, entries=150, sequenceid=371, filesize=11.9 K 2023-04-05 04:58:31,947 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/489a8793ff964358927908c277f837dd as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/489a8793ff964358927908c277f837dd 2023-04-05 04:58:31,950 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/489a8793ff964358927908c277f837dd, entries=150, sequenceid=371, filesize=11.9 K 2023-04-05 04:58:31,950 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/e8acabe08ee94a418d80d3d61ae7824d as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/e8acabe08ee94a418d80d3d61ae7824d 2023-04-05 04:58:31,953 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/e8acabe08ee94a418d80d3d61ae7824d, entries=150, sequenceid=371, filesize=11.9 K 2023-04-05 04:58:31,954 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for 3fee67ac8cf8f50b334cb2fd8fafdea1 in 841ms, sequenceid=371, compaction requested=true 2023-04-05 04:58:31,954 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:31,954 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:31,954 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:31,954 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:31,954 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,954 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:31,954 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:31,955 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,955 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:31,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:31,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:31,955 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:31,955 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:31,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:31,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:31,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:31,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:31,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:31,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3be7d13[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:31,956 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:31,957 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:31,957 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:31,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:31,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:31,957 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:31,957 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:31,957 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,957 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:31,957 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:31,958 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,958 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:31,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:31,958 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:31,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:31,958 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:31,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:31,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:31,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:31,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:31,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:31,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:31,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:31,960 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:31,960 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:31,960 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:31,960 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:31,960 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:31,961 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:31,961 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:31,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:31,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:31,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:31,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:31,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:31,961 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:31,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:31,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:31,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:31,961 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:31,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:31,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:31,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:31,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,962 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:31,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:31,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,964 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,964 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:31,964 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:31,964 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:31,964 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:31,964 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:31,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:31,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:31,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:31,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:31,964 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:31,964 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:31,964 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:31,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:31,964 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:31,965 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:31,965 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:31,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:31,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:32,065 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:32,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:32,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:32,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:32,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:32,068 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:32,068 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:32,068 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:32,068 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:32,068 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,068 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,069 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:32,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:32,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:32,069 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,069 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,069 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:32,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,070 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:32,070 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:32,070 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:32,070 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:32,070 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:32,070 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:32,070 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:32,070 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:32,070 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:32,070 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:32,070 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3fee67ac8cf8f50b334cb2fd8fafdea1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 04:58:32,070 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=A 2023-04-05 04:58:32,071 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:32,071 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=B 2023-04-05 04:58:32,071 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:32,071 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=C 2023-04-05 04:58:32,071 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:32,080 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=381 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/45f59a87750a4fbb94567fc0c07deaaf 2023-04-05 04:58:32,091 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=381 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/36cf7eca82b34e5d8337e796bf244cb5 2023-04-05 04:58:32,100 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=381 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/7c82f5b1dced4b4fb1dc3f0f1aabbb01 2023-04-05 04:58:32,105 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/45f59a87750a4fbb94567fc0c07deaaf as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/45f59a87750a4fbb94567fc0c07deaaf 2023-04-05 04:58:32,108 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/45f59a87750a4fbb94567fc0c07deaaf, entries=150, sequenceid=381, filesize=11.9 K 2023-04-05 04:58:32,109 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/36cf7eca82b34e5d8337e796bf244cb5 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/36cf7eca82b34e5d8337e796bf244cb5 2023-04-05 04:58:32,112 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/36cf7eca82b34e5d8337e796bf244cb5, entries=150, sequenceid=381, filesize=11.9 K 2023-04-05 04:58:32,112 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/7c82f5b1dced4b4fb1dc3f0f1aabbb01 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/7c82f5b1dced4b4fb1dc3f0f1aabbb01 2023-04-05 04:58:32,115 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/7c82f5b1dced4b4fb1dc3f0f1aabbb01, entries=150, sequenceid=381, filesize=11.9 K 2023-04-05 04:58:32,115 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for 3fee67ac8cf8f50b334cb2fd8fafdea1 in 45ms, sequenceid=381, compaction requested=true 2023-04-05 04:58:32,115 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:32,115 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:32,116 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:32,116 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:32,116 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,116 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:32,116 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:32,116 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,116 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:32,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:32,117 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:32,117 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:32,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:32,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:32,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:32,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54cf6ec9[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:32,118 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:32,118 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,119 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,119 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:32,119 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:32,119 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:32,120 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,120 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:32,121 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:32,121 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:32,121 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,121 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:32,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:32,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:32,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:32,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:32,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:32,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:32,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:32,123 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:32,123 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:32,123 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:32,123 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:32,124 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:32,124 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,124 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:32,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:32,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,124 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:32,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:32,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:32,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:32,125 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:32,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,127 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:32,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,138 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,138 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,138 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,138 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,138 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,138 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,138 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,138 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:32,138 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:32,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:32,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:32,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:32,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:32,138 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:32,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:32,138 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:32,138 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:32,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:32,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:32,238 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:32,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:32,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:32,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:32,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:32,241 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:32,241 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:32,241 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:32,241 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:32,241 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,241 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,242 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:32,242 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:32,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:32,242 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,242 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:32,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:32,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,243 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:32,243 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:32,243 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:32,243 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:32,243 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:32,243 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:32,243 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:32,243 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:32,243 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:32,243 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:32,243 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:32,243 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:32,244 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:32,244 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:32,244 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,244 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:32,244 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:32,245 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,245 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:32,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:32,245 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,245 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:32,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:32,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:32,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:32,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:32,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c397bef[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:32,247 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:32,247 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,248 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,248 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:32,248 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:32,248 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,248 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:32,248 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:32,249 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,249 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:32,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:32,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:32,249 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:32,249 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:32,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:32,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:32,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:32,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:32,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:32,251 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:32,251 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:32,251 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:32,251 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:32,251 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:32,251 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,252 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:32,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:32,252 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:32,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:32,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:32,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:32,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,253 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:32,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,254 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:32,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,255 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,255 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:32,255 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,255 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:32,255 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:32,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:32,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:32,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:32,255 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:32,256 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:32,256 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,256 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:32,256 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,256 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,256 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:32,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:32,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 3fee67ac8cf8f50b334cb2fd8fafdea1 2023-04-05 04:58:32,313 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3fee67ac8cf8f50b334cb2fd8fafdea1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 04:58:32,313 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=A 2023-04-05 04:58:32,313 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:32,313 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=B 2023-04-05 04:58:32,313 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:32,313 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=C 2023-04-05 04:58:32,313 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:32,351 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:32,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59330 deadline: 1680670772345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,351 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:32,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59304 deadline: 1680670772348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670712336, max timestamp=1680670712350Num uniques 859; 2023-04-05 04:58:32,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:58:32,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670712336, max timestamp=1680670712350Num uniques 859; 2023-04-05 04:58:32,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670712336, max timestamp=1680670712350Num uniques 859; 2023-04-05 04:58:32,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:58:32,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:58:32,356 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:32,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:32,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:32,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:32,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:32,359 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:32,359 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:32,359 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:32,359 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:32,359 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,359 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,360 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,360 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:32,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:32,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:32,360 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,360 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:32,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:32,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,361 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:32,361 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:32,361 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:32,361 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:32,361 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:32,361 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:32,361 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:32,361 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:32,361 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:32,361 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:32,361 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:32,361 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:32,361 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:32,361 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:32,362 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,362 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:32,362 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:32,362 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,362 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:32,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:32,363 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,363 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:32,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:32,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:32,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:32,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:32,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dfb5fe8[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:32,364 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:32,364 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,364 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,364 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:32,364 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:32,364 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,364 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:32,364 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:32,365 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,365 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:32,365 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:32,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:32,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:32,365 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:32,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:32,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:32,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:32,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:32,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:32,367 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:32,367 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:32,367 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:32,367 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:32,367 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:32,368 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,368 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,368 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:32,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:32,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:32,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:32,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:32,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:32,370 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:32,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,371 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:32,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,372 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,372 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:32,372 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,372 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:32,372 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:32,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:32,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:32,372 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:32,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:32,372 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,372 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,372 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,372 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,372 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:32,372 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:32,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:32,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:32,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:32,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59314 deadline: 1680670772448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,461 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:32,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59330 deadline: 1680670772452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,461 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:32,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59304 deadline: 1680670772452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,472 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:32,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:32,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:32,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:32,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:32,475 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:32,475 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:32,475 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:32,475 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:32,475 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,476 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,476 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,476 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:32,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:32,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:32,476 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,476 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:32,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:32,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,477 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:32,477 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:32,477 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:32,477 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:32,477 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:32,478 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:32,478 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:32,478 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:32,478 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:32,478 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:32,478 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:32,478 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:32,478 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:32,478 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:32,478 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,478 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:32,478 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:32,479 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:32,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:32,479 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:32,479 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,479 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:32,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:32,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:32,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:32,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d5de1a1[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:32,480 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:32,480 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,481 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,481 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:32,481 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,481 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:32,481 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:32,481 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:32,481 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:32,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:32,481 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:32,482 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:32,482 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:32,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:32,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:32,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:32,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:32,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:32,483 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:32,483 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:32,483 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:32,483 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:32,483 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:32,484 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,484 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,484 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:32,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:32,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:32,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,485 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:32,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:32,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:32,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:32,485 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:32,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,487 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,487 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,487 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:32,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:32,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:32,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:32,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:32,487 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:32,487 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:32,487 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:32,487 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:32,487 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:32,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:32,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:32,487 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,487 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,487 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,487 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,560 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:32,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59314 deadline: 1680670772555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,587 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:32,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:32,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:32,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:32,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:32,590 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:32,590 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:32,590 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:32,590 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:32,590 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,590 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,591 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,591 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:32,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:32,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:32,591 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,591 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:32,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:32,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,591 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:32,592 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:32,592 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:32,592 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:32,592 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:32,592 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:32,592 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:32,592 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:32,592 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:32,592 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:32,592 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:32,592 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:32,592 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:32,592 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:32,592 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,592 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:32,592 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:32,593 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:32,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:32,593 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:32,593 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,593 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:32,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:32,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:32,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:32,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@324b6de5[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:32,594 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:32,594 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,595 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,595 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:32,595 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:32,595 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:32,595 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,595 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:32,595 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,595 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:32,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:32,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:32,596 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:32,596 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:32,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:32,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:32,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:32,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:32,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:32,597 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:32,598 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:32,598 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:32,598 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:32,598 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:32,598 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,598 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,598 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:32,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:32,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:32,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:32,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:32,599 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:32,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:32,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,600 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:32,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,601 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,601 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:32,601 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:32,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:32,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:32,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:32,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:32,601 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,601 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:32,601 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:32,602 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:32,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:32,602 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,602 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:32,602 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:32,602 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,602 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:32,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59330 deadline: 1680670772662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,665 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:32,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59304 deadline: 1680670772663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,702 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:32,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:32,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:32,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:32,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:32,704 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:32,704 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:32,704 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:32,704 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:32,705 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,705 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,705 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:32,705 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:32,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:32,706 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,706 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:32,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:32,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,706 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:32,706 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:32,706 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:32,707 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:32,707 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:32,707 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:32,707 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:32,707 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:32,707 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:32,707 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:32,707 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:32,707 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:32,707 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:32,707 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:32,707 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,707 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:32,707 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:32,708 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,708 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:32,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:32,708 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,708 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:32,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:32,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:32,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:32,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:32,710 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:32,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@352a8e40[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:32,710 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,711 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,711 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,711 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:32,711 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:32,711 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:32,711 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:32,712 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:32,712 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:32,712 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,712 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:32,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:32,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:32,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:32,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:32,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:32,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:32,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:32,714 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:32,714 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:32,714 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:32,714 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:32,714 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:32,714 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,714 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:32,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:32,714 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:32,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:32,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:32,715 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:32,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:32,716 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:32,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,717 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:32,717 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:32,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:32,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:32,717 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,717 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:32,717 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,717 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,717 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,717 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,717 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,717 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:32,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:32,717 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:32,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:32,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:32,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:32,717 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:32,718 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=393 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/8d204f9f1bb94bf888d5d93bd8a8eaf4 2023-04-05 04:58:32,765 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:32,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59314 deadline: 1680670772762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,818 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:32,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:32,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:32,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:32,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:32,820 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:32,821 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:32,821 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:32,821 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:32,821 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,821 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,822 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:32,822 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:32,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:32,822 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,822 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:32,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:32,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,822 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:32,823 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:32,823 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:32,823 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:32,823 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:32,823 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:32,823 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:32,823 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:32,823 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:32,823 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:32,823 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:32,823 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:32,823 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:32,823 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:32,823 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,823 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:32,823 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:32,824 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,824 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:32,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:32,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:32,824 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,824 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:32,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:32,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:32,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:32,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2998312[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:32,825 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:32,825 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,826 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,826 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:32,826 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:32,826 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,826 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:32,826 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:32,827 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,827 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:32,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:32,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:32,827 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:32,827 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:32,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:32,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:32,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:32,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:32,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:32,829 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:32,829 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:32,829 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:32,829 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:32,829 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:32,830 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:32,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:32,830 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,830 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:32,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:32,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:32,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:32,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:32,833 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:32,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,834 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,835 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,836 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:32,836 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:32,836 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:32,836 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:32,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:32,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:32,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:32,836 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,836 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,836 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,836 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,836 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:32,836 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,836 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:32,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:32,936 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:32,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:32,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:32,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:32,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:32,946 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:32,946 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:32,946 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:32,946 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:32,946 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,946 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,947 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,947 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:32,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:32,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:32,947 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,947 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:32,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:32,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,949 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:32,949 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:32,949 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:32,949 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:32,949 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:32,950 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:32,950 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:32,950 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:32,950 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:32,950 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:32,950 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:32,950 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:32,950 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:32,950 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:32,950 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,950 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:32,950 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:32,951 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,951 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:32,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:32,951 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,951 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:32,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:32,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:32,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:32,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:32,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70bb3a09[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:32,952 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:32,953 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,953 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,953 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:32,953 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:32,953 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:32,953 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,953 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:32,954 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:32,954 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:32,954 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:32,954 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:32,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:32,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:32,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:32,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:32,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:32,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:32,956 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:32,957 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:32,957 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:32,957 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:32,957 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:32,958 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,958 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:32,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:32,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,958 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:32,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,959 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:32,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:32,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:32,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:32,960 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:32,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:32,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,961 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,961 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,961 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:32,961 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:32,961 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,961 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,961 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:32,961 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:32,961 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:32,962 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:32,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:32,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:32,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:32,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:32,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:32,962 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:32,962 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:32,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:32,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:32,970 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:32,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59304 deadline: 1680670772965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:32,971 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:32,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59330 deadline: 1680670772966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,062 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:33,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:33,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:33,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:33,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:33,065 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:33,065 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:33,065 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:33,065 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:33,065 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,065 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,066 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:33,066 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:33,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:33,066 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,066 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:33,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:33,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,067 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:33,067 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:33,067 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:33,067 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:33,067 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:33,067 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:33,067 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:33,067 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:33,067 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:33,068 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:33,068 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:33,068 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:33,068 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:33,068 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:33,068 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,068 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:33,068 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:33,069 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,069 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:33,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:33,069 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,069 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:33,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:33,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:33,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:33,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:33,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19f655be[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:33,070 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:33,070 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,070 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,071 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:33,071 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:33,071 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:33,071 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,071 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:33,071 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:33,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59314 deadline: 1680670773067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,071 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:33,071 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:33,071 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:33,072 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:33,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:33,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:33,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:33,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:33,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:33,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:33,074 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:33,074 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:33,074 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:33,074 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:33,074 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:33,075 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:33,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:33,076 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,076 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:33,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:33,076 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:33,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:33,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:33,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,078 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:33,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,079 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:33,079 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:33,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:33,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:33,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:33,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:33,080 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,080 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,080 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:33,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:33,080 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,080 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:33,080 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,080 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,080 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,080 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:33,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:33,080 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:33,132 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=393 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/3d98a038ae9640d187a79586dad19a46 2023-04-05 04:58:33,181 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:33,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:33,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:33,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:33,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:33,183 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:33,183 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:33,183 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:33,183 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:33,184 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,184 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,184 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:33,184 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:33,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:33,186 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,186 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:33,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:33,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,186 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:33,187 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:33,187 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:33,187 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:33,187 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:33,187 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:33,187 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:33,187 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:33,187 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:33,187 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:33,187 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:33,187 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:33,187 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:33,187 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:33,187 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,187 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:33,187 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:33,188 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,188 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:33,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:33,188 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,188 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:33,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:33,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:33,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:33,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:33,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@784f804e[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:33,189 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:33,189 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,190 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,190 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:33,190 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:33,190 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,190 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:33,190 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:33,191 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,191 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:33,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:33,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:33,191 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:33,191 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:33,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:33,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:33,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:33,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:33,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:33,193 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:33,193 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:33,193 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:33,193 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:33,193 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:33,194 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,194 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,194 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:33,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:33,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:33,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:33,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:33,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:33,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,195 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:33,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,196 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:33,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,197 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,197 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,197 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,197 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:33,197 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:33,197 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,197 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:33,197 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:33,197 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:33,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:33,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:33,197 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,198 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:33,198 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:33,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:33,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:33,298 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:33,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:33,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:33,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:33,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:33,301 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:33,301 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:33,301 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:33,301 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:33,301 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,301 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,302 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:33,302 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:33,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:33,302 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,302 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:33,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:33,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,303 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:33,303 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:33,303 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:33,303 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:33,303 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:33,303 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:33,303 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:33,303 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:33,303 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:33,303 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:33,303 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:33,303 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:33,303 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:33,303 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:33,303 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,303 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:33,303 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:33,304 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,304 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:33,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:33,304 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,304 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:33,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:33,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:33,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:33,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:33,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36000d46[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:33,306 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:33,306 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,306 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,306 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:33,306 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:33,307 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:33,307 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,307 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:33,307 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,307 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:33,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:33,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:33,307 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:33,307 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:33,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:33,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:33,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:33,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:33,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:33,309 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:33,309 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:33,309 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:33,309 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:33,309 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:33,310 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,310 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:33,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:33,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,310 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:33,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:33,310 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:33,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:33,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:33,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,311 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:33,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,313 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,313 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,313 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,313 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,313 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,313 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:33,313 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,313 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:33,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:33,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:33,313 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:33,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:33,314 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:33,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:33,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:33,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:33,314 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:33,314 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:33,414 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:33,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:33,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:33,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:33,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:33,417 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:33,417 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:33,417 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:33,417 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:33,417 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,418 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,418 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,418 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:33,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:33,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:33,418 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,418 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:33,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:33,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,419 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:33,419 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:33,419 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:33,419 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:33,419 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:33,419 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:33,419 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:33,419 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:33,419 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:33,420 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:33,420 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:33,420 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:33,420 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:33,420 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:33,420 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,420 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:33,420 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:33,421 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,421 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:33,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:33,421 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,421 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:33,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:33,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:33,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:33,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:33,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fc212fd[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:33,422 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:33,422 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,422 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,423 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:33,423 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:33,423 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:33,423 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,423 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:33,423 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:33,423 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,423 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:33,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:33,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:33,423 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:33,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:33,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:33,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:33,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:33,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:33,425 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:33,425 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:33,425 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:33,425 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:33,425 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:33,426 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,426 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,426 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:33,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:33,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:33,426 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:33,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:33,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:33,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:33,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,427 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:33,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,429 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,429 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:33,429 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:33,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:33,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:33,429 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:33,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:33,429 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:33,429 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:33,430 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:33,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:33,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:33,430 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:33,430 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,430 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,430 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,430 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,477 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:33,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59330 deadline: 1680670773472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,479 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:33,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59304 deadline: 1680670773474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,530 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:33,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:33,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:33,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:33,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:33,533 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:33,533 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:33,533 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:33,533 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:33,533 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,533 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,534 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:33,534 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:33,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:33,534 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,534 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:33,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:33,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,535 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:33,535 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:33,535 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:33,535 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:33,535 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:33,535 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:33,535 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:33,535 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:33,535 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:33,536 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:33,536 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:33,536 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:33,536 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:33,536 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:33,536 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,536 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:33,536 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:33,536 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,536 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:33,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:33,537 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,537 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:33,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:33,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:33,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:33,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:33,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3432ff1e[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:33,538 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:33,538 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,538 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,538 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:33,538 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:33,538 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:33,539 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,539 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:33,539 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:33,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:33,539 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:33,539 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:33,539 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:33,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:33,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:33,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:33,540 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=393 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/39ddcb08ed07444ebdcb1cd3a8e30b1c 2023-04-05 04:58:33,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:33,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:33,541 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:33,541 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:33,541 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:33,541 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:33,541 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:33,542 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,542 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,542 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:33,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:33,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:33,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:33,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:33,543 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:33,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,543 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/8d204f9f1bb94bf888d5d93bd8a8eaf4 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/8d204f9f1bb94bf888d5d93bd8a8eaf4 2023-04-05 04:58:33,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:33,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,544 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:33,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,546 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,546 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,546 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:33,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:33,546 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:33,546 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:33,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:33,546 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,546 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:33,546 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:33,546 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:33,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:33,546 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,546 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:33,546 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,547 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/8d204f9f1bb94bf888d5d93bd8a8eaf4, entries=100, sequenceid=393, filesize=9.5 K 2023-04-05 04:58:33,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:33,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:33,547 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/3d98a038ae9640d187a79586dad19a46 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/3d98a038ae9640d187a79586dad19a46 2023-04-05 04:58:33,550 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/3d98a038ae9640d187a79586dad19a46, entries=100, sequenceid=393, filesize=9.5 K 2023-04-05 04:58:33,550 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/39ddcb08ed07444ebdcb1cd3a8e30b1c as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/39ddcb08ed07444ebdcb1cd3a8e30b1c 2023-04-05 04:58:33,552 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/39ddcb08ed07444ebdcb1cd3a8e30b1c, entries=100, sequenceid=393, filesize=9.5 K 2023-04-05 04:58:33,553 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 3fee67ac8cf8f50b334cb2fd8fafdea1 in 1240ms, sequenceid=393, compaction requested=true 2023-04-05 04:58:33,553 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:33,553 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:58:33,553 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:58:33,553 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 04:58:33,553 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 04:58:33,553 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:58:33,555 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59134 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 04:58:33,555 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59134 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 04:58:33,555 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): 3fee67ac8cf8f50b334cb2fd8fafdea1/A is initiating minor compaction (all files) 2023-04-05 04:58:33,555 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): 3fee67ac8cf8f50b334cb2fd8fafdea1/B is initiating minor compaction (all files) 2023-04-05 04:58:33,555 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3fee67ac8cf8f50b334cb2fd8fafdea1/A in TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:33,555 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3fee67ac8cf8f50b334cb2fd8fafdea1/B in TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:33,556 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/4660535f1858438eaffd68513d7c531b, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/6f316daa848c42069d26a2a7a79a231f, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/9447931e9c4d4936946ff6e546b01a13, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/45f59a87750a4fbb94567fc0c07deaaf, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/8d204f9f1bb94bf888d5d93bd8a8eaf4] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp, totalSize=57.7 K 2023-04-05 04:58:33,556 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/4a8a3a9eb60b42a789ab4b1774340ce5, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/6a0158f455ab42749784822157f88da2, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/489a8793ff964358927908c277f837dd, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/36cf7eca82b34e5d8337e796bf244cb5, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/3d98a038ae9640d187a79586dad19a46] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp, totalSize=57.7 K 2023-04-05 04:58:33,556 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 4660535f1858438eaffd68513d7c531b, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=323, earliestPutTs=1680670708501 2023-04-05 04:58:33,556 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 4a8a3a9eb60b42a789ab4b1774340ce5, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=323, earliestPutTs=1680670708501 2023-04-05 04:58:33,556 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 6f316daa848c42069d26a2a7a79a231f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1680670709658 2023-04-05 04:58:33,556 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 6a0158f455ab42749784822157f88da2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1680670709658 2023-04-05 04:58:33,556 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 9447931e9c4d4936946ff6e546b01a13, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1680670710844 2023-04-05 04:58:33,556 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 489a8793ff964358927908c277f837dd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1680670710844 2023-04-05 04:58:33,556 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 45f59a87750a4fbb94567fc0c07deaaf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1680670711167 2023-04-05 04:58:33,557 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 36cf7eca82b34e5d8337e796bf244cb5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1680670711167 2023-04-05 04:58:33,557 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 8d204f9f1bb94bf888d5d93bd8a8eaf4, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1680670712312 2023-04-05 04:58:33,557 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 3d98a038ae9640d187a79586dad19a46, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1680670712312 2023-04-05 04:58:33,567 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): 3fee67ac8cf8f50b334cb2fd8fafdea1#B#compaction#434 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:58:33,569 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3fee67ac8cf8f50b334cb2fd8fafdea1#A#compaction#435 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:58:33,577 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/ced8742e2efc4ed289b04f5ea13fb079 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/ced8742e2efc4ed289b04f5ea13fb079 2023-04-05 04:58:33,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 3fee67ac8cf8f50b334cb2fd8fafdea1 2023-04-05 04:58:33,578 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3fee67ac8cf8f50b334cb2fd8fafdea1 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-05 04:58:33,578 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=A 2023-04-05 04:58:33,578 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:58:33,578 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=B 2023-04-05 04:58:33,578 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:58:33,578 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=C 2023-04-05 04:58:33,579 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:58:33,583 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 3fee67ac8cf8f50b334cb2fd8fafdea1/A of 3fee67ac8cf8f50b334cb2fd8fafdea1 into ced8742e2efc4ed289b04f5ea13fb079(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:58:33,583 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:33,583 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1., storeName=3fee67ac8cf8f50b334cb2fd8fafdea1/A, priority=11, startTime=1680670713553; duration=0sec 2023-04-05 04:58:33,583 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:58:33,583 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-05 04:58:33,584 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59134 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-05 04:58:33,584 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): 3fee67ac8cf8f50b334cb2fd8fafdea1/C is initiating minor compaction (all files) 2023-04-05 04:58:33,584 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3fee67ac8cf8f50b334cb2fd8fafdea1/C in TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:33,584 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/0e5e4fac52a0495aae6ebef8082b86ed, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/af5b1a9e53934d2fba504b451e699e77, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/e8acabe08ee94a418d80d3d61ae7824d, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/7c82f5b1dced4b4fb1dc3f0f1aabbb01, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/39ddcb08ed07444ebdcb1cd3a8e30b1c] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp, totalSize=57.7 K 2023-04-05 04:58:33,584 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 0e5e4fac52a0495aae6ebef8082b86ed, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=323, earliestPutTs=1680670708501 2023-04-05 04:58:33,585 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting af5b1a9e53934d2fba504b451e699e77, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1680670709658 2023-04-05 04:58:33,586 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting e8acabe08ee94a418d80d3d61ae7824d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1680670710844 2023-04-05 04:58:33,586 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 7c82f5b1dced4b4fb1dc3f0f1aabbb01, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1680670711167 2023-04-05 04:58:33,586 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 39ddcb08ed07444ebdcb1cd3a8e30b1c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1680670712312 2023-04-05 04:58:33,593 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3fee67ac8cf8f50b334cb2fd8fafdea1#C#compaction#437 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:58:33,623 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:33,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59314 deadline: 1680670773617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,646 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:33,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:33,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:33,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:33,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:33,649 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:33,649 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:33,649 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:33,649 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:33,649 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,649 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,650 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,650 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:33,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:33,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:33,650 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,650 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:33,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:33,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,650 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:33,651 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:33,651 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:33,651 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:33,651 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:33,651 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:33,651 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:33,651 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:33,651 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:33,651 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:33,651 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:33,651 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:33,651 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:33,651 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:33,651 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,651 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:33,651 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:33,652 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,652 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:33,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:33,652 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,652 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:33,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:33,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:33,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:33,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:33,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30fb4426[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:33,653 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:33,653 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,654 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,654 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:33,654 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:33,654 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:33,654 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,654 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:33,654 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,654 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:33,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:33,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:33,654 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:33,655 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:33,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:33,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:33,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:33,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:33,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:33,656 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:33,656 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:33,656 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:33,656 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:33,656 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:33,657 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,657 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,657 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:33,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:33,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:33,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:33,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:33,657 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:33,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:33,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,658 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:33,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,660 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,660 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,660 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,660 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:33,660 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:33,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:33,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:33,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:33,660 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,660 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,660 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,660 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:33,660 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:33,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:33,660 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:33,661 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:33,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:33,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:33,726 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:33,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59314 deadline: 1680670773724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,761 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:33,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:33,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:33,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:33,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:33,763 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:33,763 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:33,763 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:33,763 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:33,764 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,764 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,764 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,764 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:33,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:33,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:33,765 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,765 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:33,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,765 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:33,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,765 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:33,765 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:33,765 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:33,765 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:33,766 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:33,766 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:33,766 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:33,766 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:33,766 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:33,766 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:33,766 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:33,766 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:33,766 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:33,766 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:33,766 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,766 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:33,766 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:33,767 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,767 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:33,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:33,767 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,767 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:33,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:33,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:33,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:33,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:33,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a70542[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:33,768 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:33,768 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,768 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,769 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:33,769 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:33,769 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:33,769 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,769 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:33,769 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:33,769 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,769 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:33,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:33,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:33,769 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:33,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:33,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:33,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:33,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:33,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:33,783 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:33,783 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:33,783 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:33,783 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:33,783 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:33,792 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,792 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:33,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:33,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,792 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:33,793 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:33,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:33,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:33,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:33,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,794 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:33,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,796 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,796 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:33,796 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:33,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:33,796 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,796 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:33,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:33,796 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:33,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:33,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:33,796 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:33,796 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,796 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:33,796 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,797 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,797 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:33,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:33,897 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:33,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:33,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:33,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:33,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:33,899 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:33,899 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:33,899 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:33,899 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:33,899 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,899 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,900 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:33,900 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:33,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:33,900 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,900 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:33,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:33,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,901 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:33,901 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:33,901 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:33,901 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:33,901 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:33,901 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:33,901 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:33,901 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:33,901 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:33,902 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:33,902 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:33,902 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:33,902 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:33,902 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,902 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:33,902 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:33,903 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,903 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:33,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:33,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:33,903 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,903 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:33,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:33,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:33,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:33,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@97dba91[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:33,904 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:33,904 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,905 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,905 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:33,905 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:33,905 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:33,905 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,905 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:33,905 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,905 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:33,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:33,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:33,906 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:33,906 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:33,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:33,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:33,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:33,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:33,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:33,907 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:33,907 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:33,907 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:33,907 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:33,907 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:33,908 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,908 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,908 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:33,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:33,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:33,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:33,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:33,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:33,909 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:33,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,910 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:33,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:33,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,911 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,911 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,911 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:33,911 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:33,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:33,911 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:33,911 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:33,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:33,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:33,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:33,911 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,911 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:33,911 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:33,911 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:33,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:33,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:33,911 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:33,912 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:33,932 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:33,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59314 deadline: 1680670773928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:33,974 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/a404716244064c0193cb543ff8872551 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/a404716244064c0193cb543ff8872551 2023-04-05 04:58:33,978 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 3fee67ac8cf8f50b334cb2fd8fafdea1/B of 3fee67ac8cf8f50b334cb2fd8fafdea1 into a404716244064c0193cb543ff8872551(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:58:33,978 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:33,978 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1., storeName=3fee67ac8cf8f50b334cb2fd8fafdea1/B, priority=11, startTime=1680670713553; duration=0sec 2023-04-05 04:58:33,978 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:58:33,985 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/783093e434f74f75891b2bbc376b7180 2023-04-05 04:58:33,999 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/f326b56d87564a00b3ef120d2660da6e as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/f326b56d87564a00b3ef120d2660da6e 2023-04-05 04:58:34,003 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 3fee67ac8cf8f50b334cb2fd8fafdea1/C of 3fee67ac8cf8f50b334cb2fd8fafdea1 into f326b56d87564a00b3ef120d2660da6e(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:58:34,003 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:34,003 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1., storeName=3fee67ac8cf8f50b334cb2fd8fafdea1/C, priority=11, startTime=1680670713553; duration=0sec 2023-04-05 04:58:34,003 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:58:34,012 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:34,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:34,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:34,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:34,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:34,014 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:34,014 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:34,014 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:34,014 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:34,015 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,015 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,018 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:34,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:34,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:34,018 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,019 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,019 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:34,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:34,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,019 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:34,019 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:34,020 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:34,020 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:34,020 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:34,020 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:34,026 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:34,026 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:34,026 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:34,026 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:34,026 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:34,026 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:34,026 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:34,026 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:34,026 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,026 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:34,026 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:34,027 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,027 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:34,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:34,028 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,028 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:34,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:34,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:34,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:34,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:34,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a147eaf[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:34,031 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:34,031 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,031 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,032 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:34,032 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:34,032 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:34,032 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,032 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:34,033 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:34,033 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:34,033 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,033 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:34,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:34,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:34,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:34,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:34,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:34,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:34,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:34,038 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:34,038 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:34,038 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:34,038 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:34,038 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:34,042 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:34,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:34,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:34,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:34,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:34,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:34,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,044 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:34,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,045 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:34,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,054 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:34,055 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:34,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:34,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:34,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:34,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:34,055 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:34,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:34,055 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:34,055 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,056 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,056 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,056 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,056 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,056 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:34,056 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,056 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:34,156 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:34,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:34,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:34,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:34,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:34,159 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:34,159 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:34,159 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:34,159 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:34,159 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,159 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,160 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:34,160 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:34,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:34,160 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,160 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:34,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:34,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,160 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:34,161 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:34,161 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:34,161 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:34,161 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:34,161 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:34,161 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:34,161 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:34,161 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:34,161 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:34,161 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:34,161 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:34,161 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:34,161 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:34,161 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,161 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:34,161 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:34,163 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,163 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:34,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:34,163 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,163 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:34,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:34,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:34,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:34,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:34,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40a4c5bf[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:34,164 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:34,164 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,165 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,165 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,165 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:34,165 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:34,165 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:34,165 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:34,166 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:34,166 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,166 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:34,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:34,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:34,166 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:34,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:34,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:34,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:34,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:34,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:34,168 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:34,168 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:34,168 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:34,168 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:34,168 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:34,168 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,169 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:34,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:34,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,169 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:34,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:34,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:34,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:34,169 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:34,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,170 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:34,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,171 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:34,171 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:34,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:34,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:34,171 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:34,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:34,171 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:34,171 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:34,171 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:34,171 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,171 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,171 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,171 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,172 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,172 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:34,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:34,234 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:34,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59314 deadline: 1680670774233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,271 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:34,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:34,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:34,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:34,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:34,274 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:34,274 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:34,274 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:34,274 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:34,274 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,274 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,275 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,275 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:34,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:34,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:34,275 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,275 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:34,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:34,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:34,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:34,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:34,276 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:34,276 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:34,276 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:34,276 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:34,276 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:34,276 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:34,276 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:34,276 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:34,276 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:34,276 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:34,276 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:34,276 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,277 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:34,277 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:34,277 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,277 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:34,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:34,277 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,277 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:34,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:34,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:34,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:34,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:34,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e9745fc[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:34,279 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:34,279 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,279 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,279 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:34,279 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:34,279 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:34,279 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,279 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:34,280 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:34,280 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:34,280 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,280 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:34,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:34,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:34,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:34,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:34,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:34,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:34,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:34,282 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:34,282 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:34,282 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:34,282 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:34,282 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:34,283 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,283 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,283 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:34,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:34,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:34,283 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:34,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:34,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:34,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,284 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:34,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:34,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,285 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,285 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:34,285 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:34,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:34,285 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:34,285 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,286 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,286 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,286 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,286 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:34,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:34,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:34,287 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:34,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:34,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:34,287 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:34,287 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:34,387 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:34,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:34,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:34,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:34,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:34,390 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:34,390 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:34,390 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:34,390 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:34,390 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,390 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,391 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,391 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:34,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:34,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:34,391 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,391 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:34,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:34,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,392 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:34,392 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:34,392 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:34,392 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:34,392 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:34,392 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:34,393 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:34,393 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:34,393 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:34,393 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:34,393 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:34,393 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:34,393 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:34,393 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:34,393 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,393 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:34,393 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:34,394 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,394 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:34,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:34,394 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,394 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:34,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:34,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:34,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:34,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:34,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1adad18d[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:34,395 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:34,395 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,395 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/f6c0720342c3416e916f7ea3585aa30f 2023-04-05 04:58:34,396 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,396 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:34,398 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:34,398 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,398 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:34,398 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:34,399 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,399 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:34,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:34,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:34,399 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:34,399 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:34,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:34,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:34,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:34,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:34,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:34,402 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:34,402 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:34,402 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:34,402 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:34,402 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:34,403 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,403 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:34,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:34,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,403 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:34,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:34,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:34,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:34,404 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:34,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,406 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:34,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,408 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,408 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,408 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:34,408 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:34,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:34,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:34,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:34,408 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:34,408 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,408 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,408 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,408 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,408 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:34,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:34,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:34,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:34,409 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:34,409 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:34,411 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/e94765e9c2dd4f26bf07df58a1bb7855 2023-04-05 04:58:34,414 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/783093e434f74f75891b2bbc376b7180 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/783093e434f74f75891b2bbc376b7180 2023-04-05 04:58:34,416 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/783093e434f74f75891b2bbc376b7180, entries=200, sequenceid=430, filesize=14.3 K 2023-04-05 04:58:34,417 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/f6c0720342c3416e916f7ea3585aa30f as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/f6c0720342c3416e916f7ea3585aa30f 2023-04-05 04:58:34,419 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/f6c0720342c3416e916f7ea3585aa30f, entries=150, sequenceid=430, filesize=11.9 K 2023-04-05 04:58:34,420 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/e94765e9c2dd4f26bf07df58a1bb7855 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/e94765e9c2dd4f26bf07df58a1bb7855 2023-04-05 04:58:34,423 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/e94765e9c2dd4f26bf07df58a1bb7855, entries=150, sequenceid=430, filesize=11.9 K 2023-04-05 04:58:34,423 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 3fee67ac8cf8f50b334cb2fd8fafdea1 in 845ms, sequenceid=430, compaction requested=false 2023-04-05 04:58:34,423 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:34,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 3fee67ac8cf8f50b334cb2fd8fafdea1 2023-04-05 04:58:34,485 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3fee67ac8cf8f50b334cb2fd8fafdea1 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 04:58:34,485 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=A 2023-04-05 04:58:34,485 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:34,485 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=B 2023-04-05 04:58:34,485 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:34,485 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=C 2023-04-05 04:58:34,485 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:34,496 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/34b855b5cb074cb2981186e1b5628475 2023-04-05 04:58:34,509 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:34,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:34,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:34,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:34,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:34,511 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:34,511 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:34,511 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:34,511 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:34,512 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,512 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,512 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:34,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:34,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:34,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:34,513 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,513 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:34,513 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:34,513 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,514 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:34,513 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:34,514 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:34,514 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:34,514 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:34,514 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:34,514 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:34,514 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:34,514 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:34,514 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:34,514 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:34,514 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:34,514 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:34,514 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,514 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:34,514 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:34,515 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,515 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:34,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:34,515 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,515 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:34,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:34,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:34,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:34,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:34,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1332257f[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:34,516 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:34,516 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,517 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,517 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,517 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:34,517 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:34,517 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:34,517 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:34,518 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,518 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:34,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:34,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:34,518 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:34,518 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:34,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:34,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:34,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:34,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:34,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:34,520 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:34,520 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:34,520 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:34,520 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:34,520 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:34,521 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,521 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,521 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:34,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:34,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:34,521 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:34,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:34,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:34,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:34,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,522 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:34,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,523 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:34,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:34,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:34,524 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,524 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:34,524 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:34,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:34,524 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,524 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,524 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:34,524 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,524 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,524 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:34,524 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:34,524 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:34,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:34,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670714506, max timestamp=1680670714541Num uniques 859; 2023-04-05 04:58:34,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670714506, max timestamp=1680670714541Num uniques 859; 2023-04-05 04:58:34,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:58:34,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:58:34,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670714506, max timestamp=1680670714541Num uniques 859; 2023-04-05 04:58:34,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:58:34,605 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:34,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59330 deadline: 1680670774598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,611 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:34,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59304 deadline: 1680670774605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,624 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:34,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:34,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:34,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:34,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:34,627 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:34,627 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:34,627 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:34,627 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:34,627 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,627 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,628 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:34,628 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:34,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:34,628 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,628 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:34,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:34,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,629 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:34,629 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:34,629 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:34,629 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:34,629 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:34,629 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:34,629 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:34,629 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:34,629 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:34,629 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:34,629 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:34,629 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:34,629 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:34,629 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:34,629 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,629 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:34,629 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:34,630 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,630 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:34,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:34,630 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,631 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:34,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:34,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:34,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:34,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:34,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@755d839b[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:34,631 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:34,631 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,632 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,632 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,632 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:34,632 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:34,632 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:34,632 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:34,633 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,633 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:34,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:34,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:34,633 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:34,633 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:34,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:34,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:34,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:34,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:34,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:34,634 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:34,634 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:34,635 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:34,635 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:34,635 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:34,635 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,635 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:34,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:34,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,635 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:34,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:34,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:34,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:34,636 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:34,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,637 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:34,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,638 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,638 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,638 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,638 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,638 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:34,638 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:34,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:34,638 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,639 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,639 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:34,639 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:34,639 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:34,639 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:34,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:34,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:34,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:34,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:34,709 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:34,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59330 deadline: 1680670774706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,716 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:34,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59304 deadline: 1680670774712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,739 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:34,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:34,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:34,741 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:34,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59314 deadline: 1680670774738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:34,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:34,741 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:34,741 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:34,741 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:34,741 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:34,742 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,742 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,742 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:34,742 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:34,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:34,742 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,743 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:34,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:34,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,743 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:34,743 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:34,743 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:34,743 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:34,743 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:34,743 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:34,743 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:34,743 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:34,743 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:34,744 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:34,744 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:34,744 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:34,744 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:34,744 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:34,744 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,744 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:34,744 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:34,745 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,745 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:34,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:34,745 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,745 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:34,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:34,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:34,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:34,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:34,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31219069[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:34,746 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:34,746 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,747 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,747 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:34,747 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:34,747 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,747 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:34,747 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:34,748 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:34,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:34,748 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:34,748 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:34,748 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:34,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:34,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:34,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:34,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:34,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:34,749 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:34,749 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:34,749 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:34,749 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:34,749 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:34,750 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,750 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:34,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:34,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,750 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:34,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:34,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:34,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:34,751 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:34,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,752 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:34,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,753 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:34,753 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:34,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:34,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:34,754 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,754 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:34,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:34,754 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:34,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:34,754 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:34,754 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,754 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:34,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:34,754 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:34,754 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,754 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,754 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,854 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:34,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:34,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:34,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:34,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:34,857 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:34,857 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:34,857 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:34,857 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:34,858 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,858 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,858 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:34,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:34,858 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:34,858 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,858 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:34,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:34,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,859 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:34,859 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:34,859 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:34,859 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:34,859 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:34,859 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:34,860 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:34,860 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:34,860 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:34,860 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:34,860 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:34,860 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:34,860 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:34,860 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:34,860 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,860 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:34,860 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:34,861 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:34,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:34,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:34,862 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:34,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,862 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,863 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:34,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:34,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:34,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d3b6979[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:34,863 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:34,863 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,864 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,864 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:34,864 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:34,864 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:34,864 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,864 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:34,865 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,865 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:34,865 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:34,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:34,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:34,865 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:34,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:34,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:34,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:34,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:34,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:34,867 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:34,867 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:34,867 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:34,867 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:34,867 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:34,868 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,868 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:34,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,868 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:34,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:34,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:34,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:34,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:34,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,869 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:34,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:34,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,871 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,872 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,872 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,872 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:34,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:34,872 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:34,872 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,872 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:34,872 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:34,872 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:34,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:34,872 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,872 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:34,873 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:34,872 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:34,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:34,904 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/d4d0c28fe1414059b6a3a0872be3aad3 2023-04-05 04:58:34,913 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:34,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59330 deadline: 1680670774910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,922 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:34,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59304 deadline: 1680670774918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,973 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:34,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:34,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:34,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:34,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:34,975 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:34,975 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:34,975 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:34,975 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:34,975 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,975 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,976 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,976 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:34,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:34,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:34,976 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,976 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:34,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:34,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,977 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:34,977 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:34,977 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:34,977 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:34,977 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:34,977 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:34,977 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:34,977 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:34,977 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:34,977 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:34,977 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:34,977 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:34,977 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:34,977 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:34,977 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,977 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:34,977 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:34,978 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,978 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:34,978 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,978 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:34,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:34,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f20b27c[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:34,979 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:34,979 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,980 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,980 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:34,980 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:34,980 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:34,980 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,980 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:34,980 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:34,980 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:34,981 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:34,981 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:34,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:34,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:34,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:34,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:34,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:34,982 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:34,982 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:34,982 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:34,982 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:34,982 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:34,983 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,983 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:34,983 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:34,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:34,983 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,985 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:34,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:34,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,986 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,986 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,986 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:34,987 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:34,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:34,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:34,987 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:34,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:34,987 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:34,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:34,987 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:34,987 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:34,987 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:34,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:34,987 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,987 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:34,987 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:34,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:34,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:35,087 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:35,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:35,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:35,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:35,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:35,090 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:35,090 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:35,090 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:35,090 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:35,090 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:35,090 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:35,091 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:35,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:35,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:35,091 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:35,091 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,092 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:35,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:35,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:35,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:35,092 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:35,092 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:35,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:35,092 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:35,092 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:35,092 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:35,092 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:35,092 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:35,092 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:35,093 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:35,093 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:35,093 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:35,093 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:35,093 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:35,093 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,093 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:35,093 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:35,093 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:35,093 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:35,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:35,094 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:35,094 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:35,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:35,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:35,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:35,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:35,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:35,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66f8b758[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:35,095 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:35,095 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:35,096 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:35,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:35,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:35,096 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:35,096 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:35,096 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:35,096 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,096 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:35,097 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,097 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:35,097 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:35,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:35,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:35,097 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:35,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:35,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:35,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:35,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:35,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:35,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:35,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:35,100 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:35,100 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:35,100 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:35,100 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:35,100 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:35,103 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:35,103 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:35,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:35,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:35,103 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:35,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:35,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:35,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:35,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:35,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:35,103 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:35,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:35,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:35,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:35,104 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:35,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:35,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:35,106 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,106 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:35,106 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:35,106 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,106 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:35,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:35,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:35,106 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:35,106 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:35,106 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:35,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:35,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:35,106 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:35,106 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:35,106 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:35,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:35,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:35,114 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:35,114 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:35,215 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:35,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:35,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:35,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:35,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:35,217 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:35,217 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:35,217 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:35,217 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:35,217 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:35,218 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:35,218 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:35,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:35,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:35,218 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:35,219 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,219 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:35,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:35,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:35,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:35,219 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:35,219 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:35,219 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:35,219 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:35,219 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:35,220 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:35,220 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:35,220 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:35,220 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:35,220 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:35,220 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:35,220 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:35,220 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:35,220 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:35,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59330 deadline: 1680670775215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,220 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:35,220 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,220 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:35,220 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:35,221 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,221 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:35,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:35,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:35,221 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:35,221 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:35,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:35,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:35,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:35,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:35,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:35,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5be7c136[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:35,223 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:35,223 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:35,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:35,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59384 deadline: 1680670775220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,227 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:35,227 DEBUG [Thread-2860] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8160 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1., hostname=jenkins-hbase20.apache.org,38091,1680670569206, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 04:58:35,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:35,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:35,227 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:35,227 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:35,227 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:35,227 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,227 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:35,228 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:35,228 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,228 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:35,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:35,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:35,228 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:35,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:35,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:35,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:35,230 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:35,230 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:35,230 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:35,230 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:35,230 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:35,231 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:35,231 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:35,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:35,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:35,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:35,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:35,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:35,231 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:35,231 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:35,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:35,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:35,231 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:35,232 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59304 deadline: 1680670775223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:35,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:35,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:35,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:35,234 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,234 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:35,234 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:35,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:35,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:35,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:35,234 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:35,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:35,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:35,234 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:35,234 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:35,234 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,234 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:35,234 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:35,234 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:35,235 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:35,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:35,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:35,235 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:35,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:35,241 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:35,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59352 deadline: 1680670775239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,242 DEBUG [Thread-2854] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8179 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1., hostname=jenkins-hbase20.apache.org,38091,1680670569206, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 04:58:35,311 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/1fa327989c6346a28b3a1cfe5417ee4a 2023-04-05 04:58:35,314 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/34b855b5cb074cb2981186e1b5628475 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/34b855b5cb074cb2981186e1b5628475 2023-04-05 04:58:35,318 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/34b855b5cb074cb2981186e1b5628475, entries=150, sequenceid=444, filesize=11.9 K 2023-04-05 04:58:35,319 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/d4d0c28fe1414059b6a3a0872be3aad3 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/d4d0c28fe1414059b6a3a0872be3aad3 2023-04-05 04:58:35,321 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/d4d0c28fe1414059b6a3a0872be3aad3, entries=150, sequenceid=444, filesize=11.9 K 2023-04-05 04:58:35,322 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/1fa327989c6346a28b3a1cfe5417ee4a as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/1fa327989c6346a28b3a1cfe5417ee4a 2023-04-05 04:58:35,335 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:35,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:35,336 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/1fa327989c6346a28b3a1cfe5417ee4a, entries=150, sequenceid=444, filesize=11.9 K 2023-04-05 04:58:35,336 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 3fee67ac8cf8f50b334cb2fd8fafdea1 in 851ms, sequenceid=444, compaction requested=true 2023-04-05 04:58:35,336 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:35,336 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:58:35,337 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:58:35,337 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:58:35,337 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:58:35,337 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:58:35,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:35,338 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39816 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:58:35,338 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:58:35,338 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): 3fee67ac8cf8f50b334cb2fd8fafdea1/A is initiating minor compaction (all files) 2023-04-05 04:58:35,338 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): 3fee67ac8cf8f50b334cb2fd8fafdea1/B is initiating minor compaction (all files) 2023-04-05 04:58:35,338 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3fee67ac8cf8f50b334cb2fd8fafdea1/A in TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:35,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:35,338 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3fee67ac8cf8f50b334cb2fd8fafdea1/B in TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:35,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:35,338 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/a404716244064c0193cb543ff8872551, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/f6c0720342c3416e916f7ea3585aa30f, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/d4d0c28fe1414059b6a3a0872be3aad3] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp, totalSize=36.5 K 2023-04-05 04:58:35,338 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/ced8742e2efc4ed289b04f5ea13fb079, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/783093e434f74f75891b2bbc376b7180, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/34b855b5cb074cb2981186e1b5628475] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp, totalSize=38.9 K 2023-04-05 04:58:35,338 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:35,338 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:35,338 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:35,338 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:35,339 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting a404716244064c0193cb543ff8872551, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1680670711174 2023-04-05 04:58:35,339 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:35,339 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting f6c0720342c3416e916f7ea3585aa30f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1680670712386 2023-04-05 04:58:35,339 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:35,339 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting ced8742e2efc4ed289b04f5ea13fb079, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1680670711174 2023-04-05 04:58:35,339 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting d4d0c28fe1414059b6a3a0872be3aad3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1680670713601 2023-04-05 04:58:35,340 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,340 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:35,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:35,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:35,341 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 783093e434f74f75891b2bbc376b7180, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1680670712386 2023-04-05 04:58:35,341 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,341 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:35,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:35,342 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 34b855b5cb074cb2981186e1b5628475, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1680670713601 2023-04-05 04:58:35,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:35,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:35,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:35,343 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:35,344 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:35,344 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:35,344 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:35,344 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:35,344 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:35,346 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:35,346 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:35,346 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:35,346 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. started... 2023-04-05 04:58:35,346 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3fee67ac8cf8f50b334cb2fd8fafdea1 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-05 04:58:35,346 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=A 2023-04-05 04:58:35,346 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:58:35,347 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=B 2023-04-05 04:58:35,347 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:58:35,347 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=C 2023-04-05 04:58:35,347 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:58:35,360 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): 3fee67ac8cf8f50b334cb2fd8fafdea1#B#compaction#443 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:58:35,374 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3fee67ac8cf8f50b334cb2fd8fafdea1#A#compaction#445 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:58:35,383 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=479 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/d37a7155e44a4faeab88cef678324f0f 2023-04-05 04:58:35,386 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/b9067d0e5a2348939cfd2e30345aa62d as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/b9067d0e5a2348939cfd2e30345aa62d 2023-04-05 04:58:35,391 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3fee67ac8cf8f50b334cb2fd8fafdea1/B of 3fee67ac8cf8f50b334cb2fd8fafdea1 into b9067d0e5a2348939cfd2e30345aa62d(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:58:35,391 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:35,392 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1., storeName=3fee67ac8cf8f50b334cb2fd8fafdea1/B, priority=13, startTime=1680670715337; duration=0sec 2023-04-05 04:58:35,392 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:58:35,392 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:58:35,394 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:58:35,394 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): 3fee67ac8cf8f50b334cb2fd8fafdea1/C is initiating minor compaction (all files) 2023-04-05 04:58:35,394 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3fee67ac8cf8f50b334cb2fd8fafdea1/C in TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:35,394 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/f326b56d87564a00b3ef120d2660da6e, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/e94765e9c2dd4f26bf07df58a1bb7855, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/1fa327989c6346a28b3a1cfe5417ee4a] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp, totalSize=36.5 K 2023-04-05 04:58:35,394 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting f326b56d87564a00b3ef120d2660da6e, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1680670711174 2023-04-05 04:58:35,395 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting e94765e9c2dd4f26bf07df58a1bb7855, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1680670712386 2023-04-05 04:58:35,395 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 1fa327989c6346a28b3a1cfe5417ee4a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1680670713601 2023-04-05 04:58:35,396 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/4673d6320191487bac170b37c7ef6909 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/4673d6320191487bac170b37c7ef6909 2023-04-05 04:58:35,401 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3fee67ac8cf8f50b334cb2fd8fafdea1/A of 3fee67ac8cf8f50b334cb2fd8fafdea1 into 4673d6320191487bac170b37c7ef6909(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:58:35,401 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:35,401 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1., storeName=3fee67ac8cf8f50b334cb2fd8fafdea1/A, priority=13, startTime=1680670715336; duration=0sec 2023-04-05 04:58:35,401 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:58:35,405 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): 3fee67ac8cf8f50b334cb2fd8fafdea1#C#compaction#447 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:58:35,407 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=479 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/4bb09f88e5894bbfb749fa6453e9a30c 2023-04-05 04:58:35,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 3fee67ac8cf8f50b334cb2fd8fafdea1 2023-04-05 04:58:35,725 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. as already flushing 2023-04-05 04:58:35,756 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:35,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59314 deadline: 1680670775746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,756 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:35,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59330 deadline: 1680670775750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,756 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:35,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59304 deadline: 1680670775750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,802 DEBUG [Thread-2865] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x209faa3c to 127.0.0.1:56229 2023-04-05 04:58:35,802 DEBUG [Thread-2865] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:58:35,804 DEBUG [Thread-2869] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3c9f61fa to 127.0.0.1:56229 2023-04-05 04:58:35,804 DEBUG [Thread-2869] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:58:35,806 DEBUG [Thread-2871] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x534826c2 to 127.0.0.1:56229 2023-04-05 04:58:35,806 DEBUG [Thread-2871] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:58:35,806 DEBUG [Thread-2863] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x31fd1c1f to 127.0.0.1:56229 2023-04-05 04:58:35,806 DEBUG [Thread-2863] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:58:35,810 DEBUG [Thread-2867] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5345748a to 127.0.0.1:56229 2023-04-05 04:58:35,810 DEBUG [Thread-2867] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:58:35,813 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/2a42403cfd654d289f7b0acc733831bc as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/2a42403cfd654d289f7b0acc733831bc 2023-04-05 04:58:35,815 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=479 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/516e4a257caf4976ad9016aadb86b406 2023-04-05 04:58:35,817 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3fee67ac8cf8f50b334cb2fd8fafdea1/C of 3fee67ac8cf8f50b334cb2fd8fafdea1 into 2a42403cfd654d289f7b0acc733831bc(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:58:35,817 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:35,817 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1., storeName=3fee67ac8cf8f50b334cb2fd8fafdea1/C, priority=13, startTime=1680670715337; duration=0sec 2023-04-05 04:58:35,817 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:58:35,818 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/d37a7155e44a4faeab88cef678324f0f as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/d37a7155e44a4faeab88cef678324f0f 2023-04-05 04:58:35,821 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/d37a7155e44a4faeab88cef678324f0f, entries=150, sequenceid=479, filesize=11.9 K 2023-04-05 04:58:35,822 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/4bb09f88e5894bbfb749fa6453e9a30c as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/4bb09f88e5894bbfb749fa6453e9a30c 2023-04-05 04:58:35,825 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/4bb09f88e5894bbfb749fa6453e9a30c, entries=150, sequenceid=479, filesize=11.9 K 2023-04-05 04:58:35,825 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/516e4a257caf4976ad9016aadb86b406 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/516e4a257caf4976ad9016aadb86b406 2023-04-05 04:58:35,828 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/516e4a257caf4976ad9016aadb86b406, entries=150, sequenceid=479, filesize=11.9 K 2023-04-05 04:58:35,829 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=60.38 KB/61830 for 3fee67ac8cf8f50b334cb2fd8fafdea1 in 483ms, sequenceid=479, compaction requested=false 2023-04-05 04:58:35,829 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:35,829 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:35,829 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:35,829 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:35,829 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,829 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:35,829 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:35,830 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:35,830 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:35,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:35,830 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:35,830 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:35,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:35,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:35,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:35,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:35,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:35,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7db10b92[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:35,831 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:35,831 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:35,832 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:35,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:35,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:35,832 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:35,832 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,832 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:35,832 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:35,832 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:35,832 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:35,832 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,832 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:35,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:35,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:35,833 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:35,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:35,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:35,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:35,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:35,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:35,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:35,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:35,834 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:35,834 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:35,834 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:35,834 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:35,834 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:35,835 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:35,835 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:35,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:35,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:35,835 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:35,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:35,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:35,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:35,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:35,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:35,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:35,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:35,836 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:35,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:35,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:35,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,837 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:35,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:35,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,838 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,838 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:35,838 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:35,838 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:35,838 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:35,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:35,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:35,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:35,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:35,838 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:35,838 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:35,838 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:35,838 DEBUG [Thread-2862] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:35,838 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:35,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:35,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:35,838 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:35,838 DEBUG [Thread-2862] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:35,838 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:35,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on 3fee67ac8cf8f50b334cb2fd8fafdea1 2023-04-05 04:58:35,859 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3fee67ac8cf8f50b334cb2fd8fafdea1 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-05 04:58:35,859 DEBUG [Thread-2856] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x58ae18e2 to 127.0.0.1:56229 2023-04-05 04:58:35,859 DEBUG [Thread-2852] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06846250 to 127.0.0.1:56229 2023-04-05 04:58:35,859 DEBUG [Thread-2852] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:58:35,859 DEBUG [Thread-2856] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:58:35,859 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=A 2023-04-05 04:58:35,859 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:35,859 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=B 2023-04-05 04:58:35,859 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:35,859 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=C 2023-04-05 04:58:35,859 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:35,938 DEBUG [Thread-2862] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:35,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:36,266 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=496 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/560dda6dbb154389890abe4d969ae397 2023-04-05 04:58:36,682 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=496 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/de79e1af53c5438b859e7921ab20b107 2023-04-05 04:58:37,099 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=496 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/18aa56650edb45ea866d02b76c4a0b4c 2023-04-05 04:58:37,107 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/560dda6dbb154389890abe4d969ae397 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/560dda6dbb154389890abe4d969ae397 2023-04-05 04:58:37,113 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/560dda6dbb154389890abe4d969ae397, entries=100, sequenceid=496, filesize=9.5 K 2023-04-05 04:58:37,114 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/de79e1af53c5438b859e7921ab20b107 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/de79e1af53c5438b859e7921ab20b107 2023-04-05 04:58:37,117 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/de79e1af53c5438b859e7921ab20b107, entries=100, sequenceid=496, filesize=9.5 K 2023-04-05 04:58:37,118 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/18aa56650edb45ea866d02b76c4a0b4c as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/18aa56650edb45ea866d02b76c4a0b4c 2023-04-05 04:58:37,121 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/18aa56650edb45ea866d02b76c4a0b4c, entries=100, sequenceid=496, filesize=9.5 K 2023-04-05 04:58:37,122 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for 3fee67ac8cf8f50b334cb2fd8fafdea1 in 1263ms, sequenceid=496, compaction requested=true 2023-04-05 04:58:37,122 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:37,122 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:58:37,122 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:58:37,122 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:58:37,122 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:58:37,122 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:58:37,123 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35034 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:58:37,123 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): 3fee67ac8cf8f50b334cb2fd8fafdea1/B is initiating minor compaction (all files) 2023-04-05 04:58:37,123 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35034 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:58:37,123 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3fee67ac8cf8f50b334cb2fd8fafdea1/B in TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:37,123 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): 3fee67ac8cf8f50b334cb2fd8fafdea1/A is initiating minor compaction (all files) 2023-04-05 04:58:37,123 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/b9067d0e5a2348939cfd2e30345aa62d, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/4bb09f88e5894bbfb749fa6453e9a30c, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/de79e1af53c5438b859e7921ab20b107] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp, totalSize=34.2 K 2023-04-05 04:58:37,123 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3fee67ac8cf8f50b334cb2fd8fafdea1/A in TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:37,124 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/4673d6320191487bac170b37c7ef6909, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/d37a7155e44a4faeab88cef678324f0f, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/560dda6dbb154389890abe4d969ae397] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp, totalSize=34.2 K 2023-04-05 04:58:37,124 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting b9067d0e5a2348939cfd2e30345aa62d, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1680670713601 2023-04-05 04:58:37,124 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 4673d6320191487bac170b37c7ef6909, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1680670713601 2023-04-05 04:58:37,124 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 4bb09f88e5894bbfb749fa6453e9a30c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=479, earliestPutTs=1680670714571 2023-04-05 04:58:37,124 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting d37a7155e44a4faeab88cef678324f0f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=479, earliestPutTs=1680670714571 2023-04-05 04:58:37,124 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting de79e1af53c5438b859e7921ab20b107, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1680670715857 2023-04-05 04:58:37,124 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 560dda6dbb154389890abe4d969ae397, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1680670715857 2023-04-05 04:58:37,138 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): 3fee67ac8cf8f50b334cb2fd8fafdea1#B#compaction#453 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:58:37,138 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3fee67ac8cf8f50b334cb2fd8fafdea1#A#compaction#452 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:58:37,556 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/d7cef0d2d0bf479b8b5fe87667b93ac5 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/d7cef0d2d0bf479b8b5fe87667b93ac5 2023-04-05 04:58:37,556 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/c43a6c1160c040d080c4a5e103882f7b as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/c43a6c1160c040d080c4a5e103882f7b 2023-04-05 04:58:37,561 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3fee67ac8cf8f50b334cb2fd8fafdea1/A of 3fee67ac8cf8f50b334cb2fd8fafdea1 into d7cef0d2d0bf479b8b5fe87667b93ac5(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:58:37,561 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3fee67ac8cf8f50b334cb2fd8fafdea1/B of 3fee67ac8cf8f50b334cb2fd8fafdea1 into c43a6c1160c040d080c4a5e103882f7b(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:58:37,561 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:37,561 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:37,561 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1., storeName=3fee67ac8cf8f50b334cb2fd8fafdea1/A, priority=13, startTime=1680670717122; duration=0sec 2023-04-05 04:58:37,561 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1., storeName=3fee67ac8cf8f50b334cb2fd8fafdea1/B, priority=13, startTime=1680670717122; duration=0sec 2023-04-05 04:58:37,562 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:58:37,562 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:58:37,562 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:58:37,563 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35034 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:58:37,563 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): 3fee67ac8cf8f50b334cb2fd8fafdea1/C is initiating minor compaction (all files) 2023-04-05 04:58:37,563 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3fee67ac8cf8f50b334cb2fd8fafdea1/C in TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:37,563 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/2a42403cfd654d289f7b0acc733831bc, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/516e4a257caf4976ad9016aadb86b406, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/18aa56650edb45ea866d02b76c4a0b4c] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp, totalSize=34.2 K 2023-04-05 04:58:37,564 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 2a42403cfd654d289f7b0acc733831bc, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1680670713601 2023-04-05 04:58:37,564 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 516e4a257caf4976ad9016aadb86b406, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=479, earliestPutTs=1680670714571 2023-04-05 04:58:37,564 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 18aa56650edb45ea866d02b76c4a0b4c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1680670715857 2023-04-05 04:58:37,574 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3fee67ac8cf8f50b334cb2fd8fafdea1#C#compaction#454 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:58:37,771 DEBUG [Thread-2858] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x029c1e76 to 127.0.0.1:56229 2023-04-05 04:58:37,771 DEBUG [Thread-2858] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:58:37,988 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/a6758448d07244eabf0c1ddaef09231c as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/a6758448d07244eabf0c1ddaef09231c 2023-04-05 04:58:37,995 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3fee67ac8cf8f50b334cb2fd8fafdea1/C of 3fee67ac8cf8f50b334cb2fd8fafdea1 into a6758448d07244eabf0c1ddaef09231c(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:58:37,995 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:37,995 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1., storeName=3fee67ac8cf8f50b334cb2fd8fafdea1/C, priority=13, startTime=1680670717122; duration=0sec 2023-04-05 04:58:37,995 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:58:45,254 DEBUG [Thread-2854] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x647f95f6 to 127.0.0.1:56229 2023-04-05 04:58:45,254 DEBUG [Thread-2854] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:58:45,254 DEBUG [Thread-2860] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c240146 to 127.0.0.1:56229 2023-04-05 04:58:45,254 DEBUG [Thread-2860] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:58:45,255 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-05 04:58:45,255 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(394): wrote 79 2023-04-05 04:58:45,255 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(394): wrote 84 2023-04-05 04:58:45,255 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(394): wrote 77 2023-04-05 04:58:45,255 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(394): wrote 84 2023-04-05 04:58:45,255 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(394): wrote 83 2023-04-05 04:58:45,255 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-05 04:58:45,255 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-05 04:58:45,255 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(402): scanned 2217 2023-04-05 04:58:45,255 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(403): verified 6642 rows 2023-04-05 04:58:45,255 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(402): scanned 2242 2023-04-05 04:58:45,255 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(403): verified 6725 rows 2023-04-05 04:58:45,255 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(402): scanned 2221 2023-04-05 04:58:45,255 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(403): verified 6663 rows 2023-04-05 04:58:45,255 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(402): scanned 2228 2023-04-05 04:58:45,255 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(403): verified 6680 rows 2023-04-05 04:58:45,255 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(402): scanned 2237 2023-04-05 04:58:45,255 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(403): verified 6711 rows 2023-04-05 04:58:45,256 INFO [Listener at localhost.localdomain/34579] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-05 04:58:45,256 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x51ff7cd4 to 127.0.0.1:56229 2023-04-05 04:58:45,256 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:58:45,257 INFO [Listener at localhost.localdomain/34579] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-05 04:58:45,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-05 04:58:45,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-05 04:58:45,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-05 04:58:45,262 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680670725262"}]},"ts":"1680670725262"} 2023-04-05 04:58:45,262 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-05 04:58:45,271 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-05 04:58:45,271 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3fee67ac8cf8f50b334cb2fd8fafdea1, UNASSIGN}] 2023-04-05 04:58:45,272 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3fee67ac8cf8f50b334cb2fd8fafdea1, UNASSIGN 2023-04-05 04:58:45,273 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=3fee67ac8cf8f50b334cb2fd8fafdea1, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:45,273 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670725273"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680670725273"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680670725273"}]},"ts":"1680670725273"} 2023-04-05 04:58:45,274 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206}] 2023-04-05 04:58:45,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-05 04:58:45,426 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 3fee67ac8cf8f50b334cb2fd8fafdea1 2023-04-05 04:58:45,426 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 3fee67ac8cf8f50b334cb2fd8fafdea1, disabling compactions & flushes 2023-04-05 04:58:45,426 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:45,426 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:45,426 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. after waiting 0 ms 2023-04-05 04:58:45,427 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:45,427 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 3fee67ac8cf8f50b334cb2fd8fafdea1 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-05 04:58:45,427 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=A 2023-04-05 04:58:45,427 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:45,427 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=B 2023-04-05 04:58:45,427 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:45,427 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3fee67ac8cf8f50b334cb2fd8fafdea1, store=C 2023-04-05 04:58:45,427 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:45,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-05 04:58:45,840 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/55d28f41b8334e5787e393a32b4d4c9e 2023-04-05 04:58:45,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-05 04:58:46,260 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/242dd560aa5340c480e9d7411e271cbe 2023-04-05 04:58:46,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-05 04:58:46,677 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/ef0a60962c9749cbb3970eda5c9e3552 2023-04-05 04:58:46,681 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/A/55d28f41b8334e5787e393a32b4d4c9e as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/55d28f41b8334e5787e393a32b4d4c9e 2023-04-05 04:58:46,687 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/55d28f41b8334e5787e393a32b4d4c9e, entries=100, sequenceid=505, filesize=9.5 K 2023-04-05 04:58:46,688 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/B/242dd560aa5340c480e9d7411e271cbe as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/242dd560aa5340c480e9d7411e271cbe 2023-04-05 04:58:46,692 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/242dd560aa5340c480e9d7411e271cbe, entries=100, sequenceid=505, filesize=9.5 K 2023-04-05 04:58:46,693 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/.tmp/C/ef0a60962c9749cbb3970eda5c9e3552 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/ef0a60962c9749cbb3970eda5c9e3552 2023-04-05 04:58:46,698 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/ef0a60962c9749cbb3970eda5c9e3552, entries=100, sequenceid=505, filesize=9.5 K 2023-04-05 04:58:46,699 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 3fee67ac8cf8f50b334cb2fd8fafdea1 in 1272ms, sequenceid=505, compaction requested=false 2023-04-05 04:58:46,721 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/8351a6d639b141f8bdd8d7aa8c367ca3, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/26874a3134bc41fd8a4844a727fdf444, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/64073bfed5dc4123b223be53700b4d89, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/53e3b5735d2c48f6b5a29ad4566e8f8a, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/4a561539dea94d39a456600a86fcc651, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/9440036470844f8e898f09b8987cfceb, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/460a4fc48c494b2bae3bb5d171b8c7ac, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/bfdf4408a323482d825b966f8af4c3ff, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/687b79f194a84eaca138bb2e41122b0e, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/e42965417dd5439d88f8db83a2776640, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/7fa596a661be42b68a4581ed27d14d6c, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/c6834c986d914be0b736f42ccf9a4c86, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/434d5c9fb2514552842ee13f46abe9c3, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/a5e7f27f76974c489512a9af97626dea, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/291c444351d34c63aa187f0e07f68e74, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/fb87fedc1c454dc885dd6500bac26807, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/fca4f5cacecd4e2681a2fddff82cea1b, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/e230cebb6dc640a6b862761e38bf52ca, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/a62e2a52afa243d38b79ffee41e1f568, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/4660535f1858438eaffd68513d7c531b, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/6f316daa848c42069d26a2a7a79a231f, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/9447931e9c4d4936946ff6e546b01a13, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/45f59a87750a4fbb94567fc0c07deaaf, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/ced8742e2efc4ed289b04f5ea13fb079, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/8d204f9f1bb94bf888d5d93bd8a8eaf4, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/783093e434f74f75891b2bbc376b7180, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/4673d6320191487bac170b37c7ef6909, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/34b855b5cb074cb2981186e1b5628475, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/d37a7155e44a4faeab88cef678324f0f, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/560dda6dbb154389890abe4d969ae397] to archive 2023-04-05 04:58:46,721 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 04:58:46,722 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/8351a6d639b141f8bdd8d7aa8c367ca3 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/8351a6d639b141f8bdd8d7aa8c367ca3 2023-04-05 04:58:46,723 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/26874a3134bc41fd8a4844a727fdf444 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/26874a3134bc41fd8a4844a727fdf444 2023-04-05 04:58:46,724 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/64073bfed5dc4123b223be53700b4d89 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/64073bfed5dc4123b223be53700b4d89 2023-04-05 04:58:46,725 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/53e3b5735d2c48f6b5a29ad4566e8f8a to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/53e3b5735d2c48f6b5a29ad4566e8f8a 2023-04-05 04:58:46,725 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/4a561539dea94d39a456600a86fcc651 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/4a561539dea94d39a456600a86fcc651 2023-04-05 04:58:46,726 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/9440036470844f8e898f09b8987cfceb to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/9440036470844f8e898f09b8987cfceb 2023-04-05 04:58:46,727 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/460a4fc48c494b2bae3bb5d171b8c7ac to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/460a4fc48c494b2bae3bb5d171b8c7ac 2023-04-05 04:58:46,727 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/bfdf4408a323482d825b966f8af4c3ff to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/bfdf4408a323482d825b966f8af4c3ff 2023-04-05 04:58:46,728 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/687b79f194a84eaca138bb2e41122b0e to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/687b79f194a84eaca138bb2e41122b0e 2023-04-05 04:58:46,729 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/e42965417dd5439d88f8db83a2776640 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/e42965417dd5439d88f8db83a2776640 2023-04-05 04:58:46,729 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/7fa596a661be42b68a4581ed27d14d6c to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/7fa596a661be42b68a4581ed27d14d6c 2023-04-05 04:58:46,730 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/c6834c986d914be0b736f42ccf9a4c86 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/c6834c986d914be0b736f42ccf9a4c86 2023-04-05 04:58:46,731 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/434d5c9fb2514552842ee13f46abe9c3 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/434d5c9fb2514552842ee13f46abe9c3 2023-04-05 04:58:46,732 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/a5e7f27f76974c489512a9af97626dea to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/a5e7f27f76974c489512a9af97626dea 2023-04-05 04:58:46,733 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/291c444351d34c63aa187f0e07f68e74 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/291c444351d34c63aa187f0e07f68e74 2023-04-05 04:58:46,734 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/fb87fedc1c454dc885dd6500bac26807 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/fb87fedc1c454dc885dd6500bac26807 2023-04-05 04:58:46,735 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/fca4f5cacecd4e2681a2fddff82cea1b to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/fca4f5cacecd4e2681a2fddff82cea1b 2023-04-05 04:58:46,736 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/e230cebb6dc640a6b862761e38bf52ca to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/e230cebb6dc640a6b862761e38bf52ca 2023-04-05 04:58:46,737 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/a62e2a52afa243d38b79ffee41e1f568 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/a62e2a52afa243d38b79ffee41e1f568 2023-04-05 04:58:46,739 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/4660535f1858438eaffd68513d7c531b to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/4660535f1858438eaffd68513d7c531b 2023-04-05 04:58:46,740 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/6f316daa848c42069d26a2a7a79a231f to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/6f316daa848c42069d26a2a7a79a231f 2023-04-05 04:58:46,741 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/9447931e9c4d4936946ff6e546b01a13 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/9447931e9c4d4936946ff6e546b01a13 2023-04-05 04:58:46,742 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/45f59a87750a4fbb94567fc0c07deaaf to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/45f59a87750a4fbb94567fc0c07deaaf 2023-04-05 04:58:46,743 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/ced8742e2efc4ed289b04f5ea13fb079 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/ced8742e2efc4ed289b04f5ea13fb079 2023-04-05 04:58:46,744 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/8d204f9f1bb94bf888d5d93bd8a8eaf4 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/8d204f9f1bb94bf888d5d93bd8a8eaf4 2023-04-05 04:58:46,745 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/783093e434f74f75891b2bbc376b7180 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/783093e434f74f75891b2bbc376b7180 2023-04-05 04:58:46,746 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/4673d6320191487bac170b37c7ef6909 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/4673d6320191487bac170b37c7ef6909 2023-04-05 04:58:46,747 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/34b855b5cb074cb2981186e1b5628475 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/34b855b5cb074cb2981186e1b5628475 2023-04-05 04:58:46,748 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/d37a7155e44a4faeab88cef678324f0f to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/d37a7155e44a4faeab88cef678324f0f 2023-04-05 04:58:46,750 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/560dda6dbb154389890abe4d969ae397 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/560dda6dbb154389890abe4d969ae397 2023-04-05 04:58:46,754 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/12ce1bf78742479d9fd63ffdf97aa0ef, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/cd54f3d3bbab4691bbe134a1fd964178, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/77998cef7137475d9501dc1d9eeb790e, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/ed20c2a5895b4352aef63971b49119b5, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/3761b9ba0d044dc0ae2e22f5756bfe6d, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/bc56df46434a49b8b9477f15dc38337c, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/6070a8e7582e4ae49a661844d7e36abb, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/b3ffd557fabb4d38bc4e64199a7151c3, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/7a0059aa79404f71ae4debec48313c34, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/6c0194446a4243a498995ad40d8d452a, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/6162d88224b74d399892061a4df62e49, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/91db3c7f25044c81a871b1bb20f81fb0, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/c0a4da3b8e58446cba21ca23d8e05cfb, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/88e3467cfe374473979b1489d8f8bc4d, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/73eaeef6407b47b395ceb3ecabd5b23a, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/56141f5870f24bfc8e3f5316d73eeb82, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/470ea5f09cc54b31b64f11cf445c5430, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/dd546e465a32415bbdf2bce585d6e4d0, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/4a8a3a9eb60b42a789ab4b1774340ce5, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/929cabb18e3d48eeaeaa8389d4c6d2d4, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/6a0158f455ab42749784822157f88da2, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/489a8793ff964358927908c277f837dd, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/36cf7eca82b34e5d8337e796bf244cb5, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/a404716244064c0193cb543ff8872551, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/3d98a038ae9640d187a79586dad19a46, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/f6c0720342c3416e916f7ea3585aa30f, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/b9067d0e5a2348939cfd2e30345aa62d, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/d4d0c28fe1414059b6a3a0872be3aad3, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/4bb09f88e5894bbfb749fa6453e9a30c, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/de79e1af53c5438b859e7921ab20b107] to archive 2023-04-05 04:58:46,755 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 04:58:46,757 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/12ce1bf78742479d9fd63ffdf97aa0ef to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/12ce1bf78742479d9fd63ffdf97aa0ef 2023-04-05 04:58:46,758 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/cd54f3d3bbab4691bbe134a1fd964178 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/cd54f3d3bbab4691bbe134a1fd964178 2023-04-05 04:58:46,759 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/77998cef7137475d9501dc1d9eeb790e to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/77998cef7137475d9501dc1d9eeb790e 2023-04-05 04:58:46,760 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/ed20c2a5895b4352aef63971b49119b5 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/ed20c2a5895b4352aef63971b49119b5 2023-04-05 04:58:46,762 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/3761b9ba0d044dc0ae2e22f5756bfe6d to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/3761b9ba0d044dc0ae2e22f5756bfe6d 2023-04-05 04:58:46,763 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/bc56df46434a49b8b9477f15dc38337c to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/bc56df46434a49b8b9477f15dc38337c 2023-04-05 04:58:46,763 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/6070a8e7582e4ae49a661844d7e36abb to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/6070a8e7582e4ae49a661844d7e36abb 2023-04-05 04:58:46,764 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/b3ffd557fabb4d38bc4e64199a7151c3 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/b3ffd557fabb4d38bc4e64199a7151c3 2023-04-05 04:58:46,765 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/7a0059aa79404f71ae4debec48313c34 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/7a0059aa79404f71ae4debec48313c34 2023-04-05 04:58:46,766 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/6c0194446a4243a498995ad40d8d452a to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/6c0194446a4243a498995ad40d8d452a 2023-04-05 04:58:46,766 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/6162d88224b74d399892061a4df62e49 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/6162d88224b74d399892061a4df62e49 2023-04-05 04:58:46,767 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/91db3c7f25044c81a871b1bb20f81fb0 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/91db3c7f25044c81a871b1bb20f81fb0 2023-04-05 04:58:46,768 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/c0a4da3b8e58446cba21ca23d8e05cfb to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/c0a4da3b8e58446cba21ca23d8e05cfb 2023-04-05 04:58:46,769 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/88e3467cfe374473979b1489d8f8bc4d to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/88e3467cfe374473979b1489d8f8bc4d 2023-04-05 04:58:46,770 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/73eaeef6407b47b395ceb3ecabd5b23a to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/73eaeef6407b47b395ceb3ecabd5b23a 2023-04-05 04:58:46,771 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/56141f5870f24bfc8e3f5316d73eeb82 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/56141f5870f24bfc8e3f5316d73eeb82 2023-04-05 04:58:46,772 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/470ea5f09cc54b31b64f11cf445c5430 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/470ea5f09cc54b31b64f11cf445c5430 2023-04-05 04:58:46,773 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/dd546e465a32415bbdf2bce585d6e4d0 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/dd546e465a32415bbdf2bce585d6e4d0 2023-04-05 04:58:46,774 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/4a8a3a9eb60b42a789ab4b1774340ce5 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/4a8a3a9eb60b42a789ab4b1774340ce5 2023-04-05 04:58:46,774 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/929cabb18e3d48eeaeaa8389d4c6d2d4 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/929cabb18e3d48eeaeaa8389d4c6d2d4 2023-04-05 04:58:46,775 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/6a0158f455ab42749784822157f88da2 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/6a0158f455ab42749784822157f88da2 2023-04-05 04:58:46,776 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/489a8793ff964358927908c277f837dd to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/489a8793ff964358927908c277f837dd 2023-04-05 04:58:46,777 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/36cf7eca82b34e5d8337e796bf244cb5 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/36cf7eca82b34e5d8337e796bf244cb5 2023-04-05 04:58:46,778 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/a404716244064c0193cb543ff8872551 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/a404716244064c0193cb543ff8872551 2023-04-05 04:58:46,779 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/3d98a038ae9640d187a79586dad19a46 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/3d98a038ae9640d187a79586dad19a46 2023-04-05 04:58:46,780 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/f6c0720342c3416e916f7ea3585aa30f to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/f6c0720342c3416e916f7ea3585aa30f 2023-04-05 04:58:46,781 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/b9067d0e5a2348939cfd2e30345aa62d to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/b9067d0e5a2348939cfd2e30345aa62d 2023-04-05 04:58:46,782 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/d4d0c28fe1414059b6a3a0872be3aad3 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/d4d0c28fe1414059b6a3a0872be3aad3 2023-04-05 04:58:46,784 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/4bb09f88e5894bbfb749fa6453e9a30c to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/4bb09f88e5894bbfb749fa6453e9a30c 2023-04-05 04:58:46,785 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/de79e1af53c5438b859e7921ab20b107 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/de79e1af53c5438b859e7921ab20b107 2023-04-05 04:58:46,789 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/a2cff84b7368459e92516058c26763b7, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/da5cfbbfd19f4b678964847f1f7e98d4, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/cb68610214104ce7ba717da45d86b648, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/c839b4861b614016811bf13a1e66ce73, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/532f666e5318497b91519b504d738e47, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/8e6400aff4ae4e1e8c13ee0358ef8cb9, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/9f3f8407116f4ccca92318fe8cd9f321, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/ebc4752e67a743a885ad64933c4419c0, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/fe29fc5f4094489f9edbba46a5688220, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/d5bcda6d8f6847bd826647c3575fd6bc, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/c10ea1bc882c4df594285bc681dae059, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/2b7ca6a890f94e5bb491f34a0b1bd428, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/3141b17fbc114ac286a05a232328db98, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/1509c28b0f5c44ddb24f5edcfc438daa, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/b5e081fd4c504c1994d21018e73b9566, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/3f6fcdb60f154bb0af6803024a2f1caf, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/14e6f29ddaed4378900dd3afd76b35ad, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/99b8111047b94695b962ef2103f3f5aa, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/0e5e4fac52a0495aae6ebef8082b86ed, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/33518fc71ce247bbadc780056ca3cede, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/af5b1a9e53934d2fba504b451e699e77, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/e8acabe08ee94a418d80d3d61ae7824d, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/7c82f5b1dced4b4fb1dc3f0f1aabbb01, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/f326b56d87564a00b3ef120d2660da6e, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/39ddcb08ed07444ebdcb1cd3a8e30b1c, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/e94765e9c2dd4f26bf07df58a1bb7855, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/2a42403cfd654d289f7b0acc733831bc, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/1fa327989c6346a28b3a1cfe5417ee4a, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/516e4a257caf4976ad9016aadb86b406, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/18aa56650edb45ea866d02b76c4a0b4c] to archive 2023-04-05 04:58:46,790 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 04:58:46,791 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/a2cff84b7368459e92516058c26763b7 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/a2cff84b7368459e92516058c26763b7 2023-04-05 04:58:46,792 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/da5cfbbfd19f4b678964847f1f7e98d4 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/da5cfbbfd19f4b678964847f1f7e98d4 2023-04-05 04:58:46,794 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/cb68610214104ce7ba717da45d86b648 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/cb68610214104ce7ba717da45d86b648 2023-04-05 04:58:46,795 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/c839b4861b614016811bf13a1e66ce73 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/c839b4861b614016811bf13a1e66ce73 2023-04-05 04:58:46,796 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/532f666e5318497b91519b504d738e47 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/532f666e5318497b91519b504d738e47 2023-04-05 04:58:46,797 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/8e6400aff4ae4e1e8c13ee0358ef8cb9 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/8e6400aff4ae4e1e8c13ee0358ef8cb9 2023-04-05 04:58:46,799 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/9f3f8407116f4ccca92318fe8cd9f321 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/9f3f8407116f4ccca92318fe8cd9f321 2023-04-05 04:58:46,800 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/ebc4752e67a743a885ad64933c4419c0 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/ebc4752e67a743a885ad64933c4419c0 2023-04-05 04:58:46,801 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/fe29fc5f4094489f9edbba46a5688220 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/fe29fc5f4094489f9edbba46a5688220 2023-04-05 04:58:46,802 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/d5bcda6d8f6847bd826647c3575fd6bc to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/d5bcda6d8f6847bd826647c3575fd6bc 2023-04-05 04:58:46,803 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/c10ea1bc882c4df594285bc681dae059 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/c10ea1bc882c4df594285bc681dae059 2023-04-05 04:58:46,804 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/2b7ca6a890f94e5bb491f34a0b1bd428 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/2b7ca6a890f94e5bb491f34a0b1bd428 2023-04-05 04:58:46,805 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/3141b17fbc114ac286a05a232328db98 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/3141b17fbc114ac286a05a232328db98 2023-04-05 04:58:46,806 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/1509c28b0f5c44ddb24f5edcfc438daa to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/1509c28b0f5c44ddb24f5edcfc438daa 2023-04-05 04:58:46,807 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/b5e081fd4c504c1994d21018e73b9566 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/b5e081fd4c504c1994d21018e73b9566 2023-04-05 04:58:46,808 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/3f6fcdb60f154bb0af6803024a2f1caf to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/3f6fcdb60f154bb0af6803024a2f1caf 2023-04-05 04:58:46,809 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/14e6f29ddaed4378900dd3afd76b35ad to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/14e6f29ddaed4378900dd3afd76b35ad 2023-04-05 04:58:46,810 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/99b8111047b94695b962ef2103f3f5aa to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/99b8111047b94695b962ef2103f3f5aa 2023-04-05 04:58:46,812 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/0e5e4fac52a0495aae6ebef8082b86ed to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/0e5e4fac52a0495aae6ebef8082b86ed 2023-04-05 04:58:46,813 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/33518fc71ce247bbadc780056ca3cede to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/33518fc71ce247bbadc780056ca3cede 2023-04-05 04:58:46,814 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/af5b1a9e53934d2fba504b451e699e77 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/af5b1a9e53934d2fba504b451e699e77 2023-04-05 04:58:46,815 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/e8acabe08ee94a418d80d3d61ae7824d to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/e8acabe08ee94a418d80d3d61ae7824d 2023-04-05 04:58:46,816 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/7c82f5b1dced4b4fb1dc3f0f1aabbb01 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/7c82f5b1dced4b4fb1dc3f0f1aabbb01 2023-04-05 04:58:46,817 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/f326b56d87564a00b3ef120d2660da6e to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/f326b56d87564a00b3ef120d2660da6e 2023-04-05 04:58:46,819 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/39ddcb08ed07444ebdcb1cd3a8e30b1c to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/39ddcb08ed07444ebdcb1cd3a8e30b1c 2023-04-05 04:58:46,820 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/e94765e9c2dd4f26bf07df58a1bb7855 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/e94765e9c2dd4f26bf07df58a1bb7855 2023-04-05 04:58:46,821 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/2a42403cfd654d289f7b0acc733831bc to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/2a42403cfd654d289f7b0acc733831bc 2023-04-05 04:58:46,822 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/1fa327989c6346a28b3a1cfe5417ee4a to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/1fa327989c6346a28b3a1cfe5417ee4a 2023-04-05 04:58:46,823 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/516e4a257caf4976ad9016aadb86b406 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/516e4a257caf4976ad9016aadb86b406 2023-04-05 04:58:46,824 DEBUG [StoreCloser-TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/18aa56650edb45ea866d02b76c4a0b4c to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/18aa56650edb45ea866d02b76c4a0b4c 2023-04-05 04:58:46,828 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/recovered.edits/508.seqid, newMaxSeqId=508, maxSeqId=1 2023-04-05 04:58:46,829 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1. 2023-04-05 04:58:46,829 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 3fee67ac8cf8f50b334cb2fd8fafdea1: 2023-04-05 04:58:46,831 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 3fee67ac8cf8f50b334cb2fd8fafdea1 2023-04-05 04:58:46,832 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=3fee67ac8cf8f50b334cb2fd8fafdea1, regionState=CLOSED 2023-04-05 04:58:46,832 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670726832"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680670726832"}]},"ts":"1680670726832"} 2023-04-05 04:58:46,834 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-05 04:58:46,834 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 3fee67ac8cf8f50b334cb2fd8fafdea1, server=jenkins-hbase20.apache.org,38091,1680670569206 in 1.5590 sec 2023-04-05 04:58:46,835 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-05 04:58:46,835 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=3fee67ac8cf8f50b334cb2fd8fafdea1, UNASSIGN in 1.5630 sec 2023-04-05 04:58:46,836 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680670726835"}]},"ts":"1680670726835"} 2023-04-05 04:58:46,836 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-05 04:58:46,837 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-05 04:58:46,838 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5800 sec 2023-04-05 04:58:47,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-05 04:58:47,370 INFO [Listener at localhost.localdomain/34579] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-05 04:58:47,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-05 04:58:47,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 04:58:47,379 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 04:58:47,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-05 04:58:47,380 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 04:58:47,383 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1 2023-04-05 04:58:47,385 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A, FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B, FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C, FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/recovered.edits] 2023-04-05 04:58:47,388 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/55d28f41b8334e5787e393a32b4d4c9e to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/55d28f41b8334e5787e393a32b4d4c9e 2023-04-05 04:58:47,390 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/d7cef0d2d0bf479b8b5fe87667b93ac5 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/A/d7cef0d2d0bf479b8b5fe87667b93ac5 2023-04-05 04:58:47,393 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/242dd560aa5340c480e9d7411e271cbe to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/242dd560aa5340c480e9d7411e271cbe 2023-04-05 04:58:47,394 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/c43a6c1160c040d080c4a5e103882f7b to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/B/c43a6c1160c040d080c4a5e103882f7b 2023-04-05 04:58:47,397 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/a6758448d07244eabf0c1ddaef09231c to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/a6758448d07244eabf0c1ddaef09231c 2023-04-05 04:58:47,398 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/ef0a60962c9749cbb3970eda5c9e3552 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/C/ef0a60962c9749cbb3970eda5c9e3552 2023-04-05 04:58:47,400 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/recovered.edits/508.seqid to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1/recovered.edits/508.seqid 2023-04-05 04:58:47,400 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/3fee67ac8cf8f50b334cb2fd8fafdea1 2023-04-05 04:58:47,400 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-05 04:58:47,401 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 04:58:47,402 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-05 04:58:47,403 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-05 04:58:47,404 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 04:58:47,404 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-05 04:58:47,404 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680670727404"}]},"ts":"9223372036854775807"} 2023-04-05 04:58:47,406 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-05 04:58:47,406 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 3fee67ac8cf8f50b334cb2fd8fafdea1, NAME => 'TestAcidGuarantees,,1680670693576.3fee67ac8cf8f50b334cb2fd8fafdea1.', STARTKEY => '', ENDKEY => ''}] 2023-04-05 04:58:47,406 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-05 04:58:47,406 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680670727406"}]},"ts":"9223372036854775807"} 2023-04-05 04:58:47,407 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-05 04:58:47,408 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 04:58:47,409 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 37 msec 2023-04-05 04:58:47,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-05 04:58:47,482 INFO [Listener at localhost.localdomain/34579] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-05 04:58:47,501 INFO [Listener at localhost.localdomain/34579] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=225 (was 229), OpenFileDescriptor=543 (was 506) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=382 (was 504), ProcessCount=166 (was 166), AvailableMemoryMB=7248 (was 7178) - AvailableMemoryMB LEAK? - 2023-04-05 04:58:47,513 INFO [Listener at localhost.localdomain/34579] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225, OpenFileDescriptor=543, MaxFileDescriptor=60000, SystemLoadAverage=382, ProcessCount=166, AvailableMemoryMB=7247 2023-04-05 04:58:47,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-05 04:58:47,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-05 04:58:47,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-05 04:58:47,517 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-05 04:58:47,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-05 04:58:47,518 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-05 04:58:47,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-05 04:58:47,519 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:58:47,520 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f empty. 2023-04-05 04:58:47,520 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:58:47,520 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-05 04:58:47,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-05 04:58:47,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-05 04:58:47,927 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-05 04:58:47,928 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => ae76f0bbec3b73d99b1c36f6bc76ae4f, NAME => 'TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp 2023-04-05 04:58:48,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-05 04:58:48,336 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 04:58:48,336 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing ae76f0bbec3b73d99b1c36f6bc76ae4f, disabling compactions & flushes 2023-04-05 04:58:48,337 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:48,337 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:48,337 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. after waiting 0 ms 2023-04-05 04:58:48,337 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:48,337 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:48,337 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:58:48,342 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-05 04:58:48,344 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670728344"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680670728344"}]},"ts":"1680670728344"} 2023-04-05 04:58:48,345 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-05 04:58:48,346 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-05 04:58:48,346 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680670728346"}]},"ts":"1680670728346"} 2023-04-05 04:58:48,347 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-05 04:58:48,351 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ae76f0bbec3b73d99b1c36f6bc76ae4f, ASSIGN}] 2023-04-05 04:58:48,353 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ae76f0bbec3b73d99b1c36f6bc76ae4f, ASSIGN 2023-04-05 04:58:48,354 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=ae76f0bbec3b73d99b1c36f6bc76ae4f, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,38091,1680670569206; forceNewPlan=false, retain=false 2023-04-05 04:58:48,505 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=ae76f0bbec3b73d99b1c36f6bc76ae4f, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:48,505 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670728505"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680670728505"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680670728505"}]},"ts":"1680670728505"} 2023-04-05 04:58:48,507 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206}] 2023-04-05 04:58:48,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-05 04:58:48,665 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:48,665 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ae76f0bbec3b73d99b1c36f6bc76ae4f, NAME => 'TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.', STARTKEY => '', ENDKEY => ''} 2023-04-05 04:58:48,666 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:58:48,666 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 04:58:48,666 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:58:48,666 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:58:48,668 INFO [StoreOpener-ae76f0bbec3b73d99b1c36f6bc76ae4f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:58:48,671 DEBUG [StoreOpener-ae76f0bbec3b73d99b1c36f6bc76ae4f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A 2023-04-05 04:58:48,671 DEBUG [StoreOpener-ae76f0bbec3b73d99b1c36f6bc76ae4f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A 2023-04-05 04:58:48,671 INFO [StoreOpener-ae76f0bbec3b73d99b1c36f6bc76ae4f-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 04:58:48,672 INFO [StoreOpener-ae76f0bbec3b73d99b1c36f6bc76ae4f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ae76f0bbec3b73d99b1c36f6bc76ae4f columnFamilyName A 2023-04-05 04:58:48,673 INFO [StoreOpener-ae76f0bbec3b73d99b1c36f6bc76ae4f-1] regionserver.HStore(310): Store=ae76f0bbec3b73d99b1c36f6bc76ae4f/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 04:58:48,673 INFO [StoreOpener-ae76f0bbec3b73d99b1c36f6bc76ae4f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:58:48,674 DEBUG [StoreOpener-ae76f0bbec3b73d99b1c36f6bc76ae4f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B 2023-04-05 04:58:48,674 DEBUG [StoreOpener-ae76f0bbec3b73d99b1c36f6bc76ae4f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B 2023-04-05 04:58:48,674 INFO [StoreOpener-ae76f0bbec3b73d99b1c36f6bc76ae4f-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 04:58:48,675 INFO [StoreOpener-ae76f0bbec3b73d99b1c36f6bc76ae4f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ae76f0bbec3b73d99b1c36f6bc76ae4f columnFamilyName B 2023-04-05 04:58:48,675 INFO [StoreOpener-ae76f0bbec3b73d99b1c36f6bc76ae4f-1] regionserver.HStore(310): Store=ae76f0bbec3b73d99b1c36f6bc76ae4f/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 04:58:48,675 INFO [StoreOpener-ae76f0bbec3b73d99b1c36f6bc76ae4f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:58:48,676 DEBUG [StoreOpener-ae76f0bbec3b73d99b1c36f6bc76ae4f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C 2023-04-05 04:58:48,676 DEBUG [StoreOpener-ae76f0bbec3b73d99b1c36f6bc76ae4f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C 2023-04-05 04:58:48,676 INFO [StoreOpener-ae76f0bbec3b73d99b1c36f6bc76ae4f-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 04:58:48,676 INFO [StoreOpener-ae76f0bbec3b73d99b1c36f6bc76ae4f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ae76f0bbec3b73d99b1c36f6bc76ae4f columnFamilyName C 2023-04-05 04:58:48,677 INFO [StoreOpener-ae76f0bbec3b73d99b1c36f6bc76ae4f-1] regionserver.HStore(310): Store=ae76f0bbec3b73d99b1c36f6bc76ae4f/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 04:58:48,677 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:48,678 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:58:48,678 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:58:48,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-05 04:58:48,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:58:48,682 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-05 04:58:48,682 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened ae76f0bbec3b73d99b1c36f6bc76ae4f; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68350912, jitterRate=0.018507957458496094}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-05 04:58:48,682 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:58:48,682 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f., pid=56, masterSystemTime=1680670728658 2023-04-05 04:58:48,683 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:48,684 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:48,684 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=ae76f0bbec3b73d99b1c36f6bc76ae4f, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:48,684 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670728684"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680670728684"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680670728684"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680670728684"}]},"ts":"1680670728684"} 2023-04-05 04:58:48,685 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-05 04:58:48,686 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 in 178 msec 2023-04-05 04:58:48,686 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-05 04:58:48,686 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=ae76f0bbec3b73d99b1c36f6bc76ae4f, ASSIGN in 335 msec 2023-04-05 04:58:48,687 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-05 04:58:48,687 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680670728687"}]},"ts":"1680670728687"} 2023-04-05 04:58:48,687 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-05 04:58:48,689 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-05 04:58:48,690 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1740 sec 2023-04-05 04:58:49,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-05 04:58:49,628 INFO [Listener at localhost.localdomain/34579] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-05 04:58:49,629 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x5ddb3e63 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:58:49,638 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@50131d14, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:58:49,641 DEBUG [Listener at localhost.localdomain/34579] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:58:49,643 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38060, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:58:49,645 DEBUG [Listener at localhost.localdomain/34579] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-05 04:58:49,649 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47542, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-05 04:58:49,652 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-05 04:58:49,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-05 04:58:49,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-05 04:58:50,066 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-05 04:58:50,069 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-05 04:58:50,069 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-05 04:58:50,074 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-05 04:58:50,077 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ae76f0bbec3b73d99b1c36f6bc76ae4f, REOPEN/MOVE}] 2023-04-05 04:58:50,078 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ae76f0bbec3b73d99b1c36f6bc76ae4f, REOPEN/MOVE 2023-04-05 04:58:50,079 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=ae76f0bbec3b73d99b1c36f6bc76ae4f, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:50,079 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670730079"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680670730079"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680670730079"}]},"ts":"1680670730079"} 2023-04-05 04:58:50,080 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206}] 2023-04-05 04:58:50,232 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:58:50,234 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing ae76f0bbec3b73d99b1c36f6bc76ae4f, disabling compactions & flushes 2023-04-05 04:58:50,234 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:50,234 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:50,234 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. after waiting 0 ms 2023-04-05 04:58:50,234 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:50,239 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-05 04:58:50,240 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:50,240 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:58:50,240 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: ae76f0bbec3b73d99b1c36f6bc76ae4f to self. 2023-04-05 04:58:50,241 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:58:50,242 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=ae76f0bbec3b73d99b1c36f6bc76ae4f, regionState=CLOSED 2023-04-05 04:58:50,242 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670730242"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680670730242"}]},"ts":"1680670730242"} 2023-04-05 04:58:50,246 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-05 04:58:50,246 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 in 164 msec 2023-04-05 04:58:50,247 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=ae76f0bbec3b73d99b1c36f6bc76ae4f, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,38091,1680670569206; forceNewPlan=false, retain=true 2023-04-05 04:58:50,398 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=ae76f0bbec3b73d99b1c36f6bc76ae4f, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:50,398 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670730398"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680670730398"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680670730398"}]},"ts":"1680670730398"} 2023-04-05 04:58:50,399 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206}] 2023-04-05 04:58:50,555 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:50,555 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ae76f0bbec3b73d99b1c36f6bc76ae4f, NAME => 'TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.', STARTKEY => '', ENDKEY => ''} 2023-04-05 04:58:50,555 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:58:50,555 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-05 04:58:50,555 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:58:50,555 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:58:50,557 INFO [StoreOpener-ae76f0bbec3b73d99b1c36f6bc76ae4f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:58:50,557 DEBUG [StoreOpener-ae76f0bbec3b73d99b1c36f6bc76ae4f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A 2023-04-05 04:58:50,558 DEBUG [StoreOpener-ae76f0bbec3b73d99b1c36f6bc76ae4f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A 2023-04-05 04:58:50,558 INFO [StoreOpener-ae76f0bbec3b73d99b1c36f6bc76ae4f-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 04:58:50,558 INFO [StoreOpener-ae76f0bbec3b73d99b1c36f6bc76ae4f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ae76f0bbec3b73d99b1c36f6bc76ae4f columnFamilyName A 2023-04-05 04:58:50,560 INFO [StoreOpener-ae76f0bbec3b73d99b1c36f6bc76ae4f-1] regionserver.HStore(310): Store=ae76f0bbec3b73d99b1c36f6bc76ae4f/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 04:58:50,561 INFO [StoreOpener-ae76f0bbec3b73d99b1c36f6bc76ae4f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:58:50,562 DEBUG [StoreOpener-ae76f0bbec3b73d99b1c36f6bc76ae4f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B 2023-04-05 04:58:50,562 DEBUG [StoreOpener-ae76f0bbec3b73d99b1c36f6bc76ae4f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B 2023-04-05 04:58:50,562 INFO [StoreOpener-ae76f0bbec3b73d99b1c36f6bc76ae4f-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 04:58:50,562 INFO [StoreOpener-ae76f0bbec3b73d99b1c36f6bc76ae4f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ae76f0bbec3b73d99b1c36f6bc76ae4f columnFamilyName B 2023-04-05 04:58:50,563 INFO [StoreOpener-ae76f0bbec3b73d99b1c36f6bc76ae4f-1] regionserver.HStore(310): Store=ae76f0bbec3b73d99b1c36f6bc76ae4f/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 04:58:50,563 INFO [StoreOpener-ae76f0bbec3b73d99b1c36f6bc76ae4f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:58:50,565 DEBUG [StoreOpener-ae76f0bbec3b73d99b1c36f6bc76ae4f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C 2023-04-05 04:58:50,565 DEBUG [StoreOpener-ae76f0bbec3b73d99b1c36f6bc76ae4f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C 2023-04-05 04:58:50,565 INFO [StoreOpener-ae76f0bbec3b73d99b1c36f6bc76ae4f-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-05 04:58:50,566 INFO [StoreOpener-ae76f0bbec3b73d99b1c36f6bc76ae4f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ae76f0bbec3b73d99b1c36f6bc76ae4f columnFamilyName C 2023-04-05 04:58:50,566 INFO [StoreOpener-ae76f0bbec3b73d99b1c36f6bc76ae4f-1] regionserver.HStore(310): Store=ae76f0bbec3b73d99b1c36f6bc76ae4f/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-05 04:58:50,567 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:50,568 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:58:50,569 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:58:50,572 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-05 04:58:50,573 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:58:50,574 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened ae76f0bbec3b73d99b1c36f6bc76ae4f; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69122361, jitterRate=0.030003443360328674}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-05 04:58:50,574 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:58:50,575 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f., pid=61, masterSystemTime=1680670730551 2023-04-05 04:58:50,576 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:50,577 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:50,577 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=ae76f0bbec3b73d99b1c36f6bc76ae4f, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:50,577 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670730577"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680670730577"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680670730577"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680670730577"}]},"ts":"1680670730577"} 2023-04-05 04:58:50,579 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-05 04:58:50,579 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 in 179 msec 2023-04-05 04:58:50,580 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-05 04:58:50,580 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=ae76f0bbec3b73d99b1c36f6bc76ae4f, REOPEN/MOVE in 502 msec 2023-04-05 04:58:50,582 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-05 04:58:50,582 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 507 msec 2023-04-05 04:58:50,583 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 930 msec 2023-04-05 04:58:50,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-05 04:58:50,585 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x630ea023 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:58:50,589 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6722b207, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:58:50,590 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x5ec98d63 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:58:50,596 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@48502121, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:58:50,597 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x1ae3ec16 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:58:50,605 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8d3d8bc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:58:50,605 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x7bec0400 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:58:50,613 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e6ad437, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:58:50,613 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x34b8bf14 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:58:50,621 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2f6f7187, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:58:50,622 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x49c34fbe to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:58:50,634 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d74fc48, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:58:50,634 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x5467be78 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:58:50,638 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70ab5f5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:58:50,639 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x44535519 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:58:50,651 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2daeb795, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:58:50,651 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x7689435c to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:58:50,656 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28ac2b09, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:58:50,656 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(139): Connect 0x475ee808 to 127.0.0.1:56229 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-05 04:58:50,671 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7bf1c01c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-05 04:58:50,680 DEBUG [hconnection-0x71d00ac7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:58:50,680 DEBUG [hconnection-0x6cf8c3b4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:58:50,683 DEBUG [hconnection-0x118ec2f1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:58:50,684 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38086, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:58:50,686 DEBUG [hconnection-0x614aafb9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:58:50,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:50,691 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38078, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:58:50,691 DEBUG [hconnection-0x6c56b2d1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:58:50,691 DEBUG [hconnection-0x12e355b2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:58:50,694 DEBUG [hconnection-0x3388b069-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:58:50,695 DEBUG [hconnection-0x20d5947f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:58:50,695 DEBUG [hconnection-0x7cc42206-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:58:50,694 DEBUG [hconnection-0x19831015-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-05 04:58:50,695 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38064, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:58:50,700 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38096, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:58:50,700 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38100, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:58:50,700 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38102, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:58:50,703 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38118, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:58:50,705 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38124, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:58:50,705 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38126, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:58:50,705 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38120, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-05 04:58:50,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:50,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:50,709 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:50,710 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:50,710 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:50,710 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:50,714 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:50,714 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:50,715 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:50,715 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:50,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:50,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:50,716 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:50,716 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:50,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:50,716 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:50,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:50,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:50,717 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:50,717 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:50,717 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:50,717 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:50,717 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:50,717 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:50,717 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:50,717 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:50,717 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:50,718 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:58:50,719 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ae76f0bbec3b73d99b1c36f6bc76ae4f 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-05 04:58:50,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:58:50,720 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:58:50,720 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=A 2023-04-05 04:58:50,720 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:50,720 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=B 2023-04-05 04:58:50,720 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:50,720 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=C 2023-04-05 04:58:50,720 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:50,738 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:50,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38124 deadline: 1680670790733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:50,739 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670730722, max timestamp=1680670730737Num uniques 859; 2023-04-05 04:58:50,739 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:58:50,739 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670730722, max timestamp=1680670730737Num uniques 859; 2023-04-05 04:58:50,739 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:58:50,739 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670730722, max timestamp=1680670730737Num uniques 859; 2023-04-05 04:58:50,739 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:58:50,746 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405e990361585a04399992dcad70329aa20 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405e990361585a04399992dcad70329aa20 2023-04-05 04:58:50,757 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:50,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38086 deadline: 1680670790753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:50,757 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:50,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38100 deadline: 1680670790756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:50,757 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:50,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38120 deadline: 1680670790756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:50,758 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:50,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38064 deadline: 1680670790757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:50,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:50,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38124 deadline: 1680670790840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:50,859 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:50,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38086 deadline: 1680670790858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:50,859 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:50,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38100 deadline: 1680670790858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:50,859 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:50,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38120 deadline: 1680670790858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:50,860 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:50,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38064 deadline: 1680670790859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:51,043 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:51,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38124 deadline: 1680670791042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:51,061 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:51,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38100 deadline: 1680670791060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:51,062 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:51,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38086 deadline: 1680670791061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:51,062 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:51,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38120 deadline: 1680670791061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:51,062 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:51,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38064 deadline: 1680670791062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:51,153 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=14, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/d5ec479c08424a7b9783c4833718aaf7 2023-04-05 04:58:51,161 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/c9b802fd70b14ef5b9ff33b9fe31ebfb 2023-04-05 04:58:51,346 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:51,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38124 deadline: 1680670791346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:51,363 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:51,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38100 deadline: 1680670791362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:51,364 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:51,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38086 deadline: 1680670791364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:51,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:51,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38120 deadline: 1680670791365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:51,366 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:51,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38064 deadline: 1680670791365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:51,580 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/c32fca2ae9eb49a99a7cc2e9d251ebbd 2023-04-05 04:58:51,582 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/d5ec479c08424a7b9783c4833718aaf7 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/d5ec479c08424a7b9783c4833718aaf7 2023-04-05 04:58:51,585 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/d5ec479c08424a7b9783c4833718aaf7, entries=100, sequenceid=14, filesize=18.4 K 2023-04-05 04:58:51,586 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/c9b802fd70b14ef5b9ff33b9fe31ebfb as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/c9b802fd70b14ef5b9ff33b9fe31ebfb 2023-04-05 04:58:51,589 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/c9b802fd70b14ef5b9ff33b9fe31ebfb, entries=100, sequenceid=14, filesize=9.3 K 2023-04-05 04:58:51,590 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/c32fca2ae9eb49a99a7cc2e9d251ebbd as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/c32fca2ae9eb49a99a7cc2e9d251ebbd 2023-04-05 04:58:51,594 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/c32fca2ae9eb49a99a7cc2e9d251ebbd, entries=100, sequenceid=14, filesize=9.3 K 2023-04-05 04:58:51,595 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for ae76f0bbec3b73d99b1c36f6bc76ae4f in 876ms, sequenceid=14, compaction requested=false 2023-04-05 04:58:51,595 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-05 04:58:51,595 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:58:51,595 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:51,596 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:51,596 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:51,596 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:51,596 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:51,596 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:51,597 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:51,597 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:51,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:51,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:51,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:51,597 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:51,597 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:51,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:51,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:51,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:51,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:51,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:51,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:51,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f8f50a[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:51,598 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:51,598 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:51,599 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:51,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:51,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:51,599 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:51,599 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:51,599 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:51,599 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:51,599 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:51,599 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:51,599 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:51,599 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:51,599 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:51,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:51,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:51,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:51,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:51,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:51,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:51,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:51,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:51,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:51,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:51,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:51,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:51,602 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:51,602 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:51,602 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:51,602 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:51,602 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:51,603 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:51,603 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:51,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:51,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:51,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:51,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:51,603 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:51,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:51,604 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:51,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:51,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:51,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:51,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:51,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:51,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:51,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:51,609 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:51,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:51,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:51,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:51,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:51,629 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:51,629 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:51,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:51,629 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:51,629 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:51,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:51,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:51,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:51,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:51,629 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:51,629 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:51,629 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:51,629 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:51,629 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:51,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:51,629 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:51,629 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:51,630 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:51,730 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:51,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:51,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:51,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:51,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:51,732 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:51,732 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:51,732 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:51,732 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:51,733 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:51,733 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:51,733 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:51,733 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:51,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:51,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:51,734 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:51,734 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:51,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:51,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:51,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:51,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:51,734 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:51,734 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:51,734 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:51,734 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:51,734 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:51,734 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:51,735 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:51,735 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:51,735 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:51,735 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:58:51,735 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ae76f0bbec3b73d99b1c36f6bc76ae4f 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-05 04:58:51,735 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=A 2023-04-05 04:58:51,735 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:58:51,735 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=B 2023-04-05 04:58:51,735 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:58:51,735 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=C 2023-04-05 04:58:51,735 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:58:51,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:58:51,852 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:58:51,865 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:51,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38100 deadline: 1680670791863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:51,866 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:51,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38124 deadline: 1680670791865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:51,869 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:51,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38064 deadline: 1680670791867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:51,869 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:51,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38086 deadline: 1680670791869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:51,870 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:51,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38120 deadline: 1680670791869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:51,969 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:51,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38124 deadline: 1680670791967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:52,154 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405995e368dec0a42639dd3162662f223ff to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405995e368dec0a42639dd3162662f223ff 2023-04-05 04:58:52,171 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:52,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38124 deadline: 1680670792170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:52,356 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-05 04:58:52,473 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:52,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38124 deadline: 1680670792473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:52,558 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/1c9cc481f75f4df48f7ad6a75cd2fc0c 2023-04-05 04:58:52,571 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/62c669d8076a43ab831e658745683c4b 2023-04-05 04:58:52,581 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/6c45d31e0a9040189de6c478d88c3448 2023-04-05 04:58:52,584 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/1c9cc481f75f4df48f7ad6a75cd2fc0c as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/1c9cc481f75f4df48f7ad6a75cd2fc0c 2023-04-05 04:58:52,588 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/1c9cc481f75f4df48f7ad6a75cd2fc0c, entries=150, sequenceid=50, filesize=25.1 K 2023-04-05 04:58:52,589 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/62c669d8076a43ab831e658745683c4b as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/62c669d8076a43ab831e658745683c4b 2023-04-05 04:58:52,592 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/62c669d8076a43ab831e658745683c4b, entries=150, sequenceid=50, filesize=11.6 K 2023-04-05 04:58:52,595 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/6c45d31e0a9040189de6c478d88c3448 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/6c45d31e0a9040189de6c478d88c3448 2023-04-05 04:58:52,598 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/6c45d31e0a9040189de6c478d88c3448, entries=150, sequenceid=50, filesize=11.6 K 2023-04-05 04:58:52,598 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for ae76f0bbec3b73d99b1c36f6bc76ae4f in 863ms, sequenceid=50, compaction requested=false 2023-04-05 04:58:52,598 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:58:52,599 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:52,599 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:52,599 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:52,600 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:52,601 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:52,602 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:52,603 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:52,603 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:52,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:52,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:52,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:52,603 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:52,603 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:52,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:52,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:52,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:52,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:52,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:52,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:52,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2419c0da[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:52,604 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:52,604 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:52,605 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:52,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:52,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:52,605 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:52,605 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:52,605 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:52,605 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:52,605 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:52,606 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:52,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:52,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:52,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:52,606 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:52,606 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:52,606 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:52,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:52,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:52,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:52,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:52,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:52,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:52,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:52,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:52,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:52,608 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:52,608 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:52,608 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:52,608 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:52,608 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:52,609 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:52,609 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:52,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:52,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:52,609 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:52,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:52,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:52,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:52,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:52,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:52,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:52,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:52,610 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:52,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:52,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:52,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:52,611 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:52,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:52,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:52,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:52,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:52,613 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:52,613 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:52,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:52,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:52,613 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:52,613 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:52,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:52,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:52,613 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:52,614 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:52,614 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:52,614 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:52,614 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:52,614 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:52,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:52,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:52,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:52,614 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:52,614 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:52,714 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:52,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:52,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:52,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:52,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:52,717 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:52,717 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:52,717 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:52,717 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:52,717 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:52,717 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:52,718 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:52,718 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:52,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:52,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:52,718 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:52,718 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:52,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:52,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:52,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:52,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:52,719 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:52,719 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:52,719 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:52,719 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:52,719 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:52,719 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:52,719 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:52,719 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:52,719 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:52,719 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:58:52,720 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ae76f0bbec3b73d99b1c36f6bc76ae4f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 04:58:52,720 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=A 2023-04-05 04:58:52,720 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:52,720 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=B 2023-04-05 04:58:52,720 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:52,720 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=C 2023-04-05 04:58:52,720 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:52,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:58:52,874 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:58:52,892 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670732886, max timestamp=1680670732890Num uniques 859; 2023-04-05 04:58:52,892 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670732887, max timestamp=1680670732890Num uniques 859; 2023-04-05 04:58:52,892 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:58:52,892 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:58:52,892 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670732887, max timestamp=1680670732890Num uniques 859; 2023-04-05 04:58:52,892 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:58:52,908 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:52,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38064 deadline: 1680670792905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:52,910 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:52,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38086 deadline: 1680670792907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:52,911 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:52,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38100 deadline: 1680670792908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:52,911 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:52,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38120 deadline: 1680670792908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:52,978 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:52,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38124 deadline: 1680670792977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:53,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:53,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38064 deadline: 1680670793009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:53,012 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:53,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38086 deadline: 1680670793011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:53,013 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:53,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38100 deadline: 1680670793011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:53,013 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:53,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38120 deadline: 1680670793012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:53,132 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040578d374be73464dbdbc1fc502ca613029 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040578d374be73464dbdbc1fc502ca613029 2023-04-05 04:58:53,135 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=60, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/4f27228c411a4f1b972624702521872e 2023-04-05 04:58:53,213 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:53,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38064 deadline: 1680670793211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:53,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:53,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38100 deadline: 1680670793214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:53,216 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:53,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38086 deadline: 1680670793214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:53,216 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:53,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38120 deadline: 1680670793215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:53,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:53,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38064 deadline: 1680670793514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:53,518 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:53,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38100 deadline: 1680670793516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:53,518 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:53,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38120 deadline: 1680670793517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:53,519 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:53,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38086 deadline: 1680670793517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:53,543 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/ff9381ebcd8749cd8e07ab5d55e75da9 2023-04-05 04:58:53,951 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/701b0d86095b4c94ae1b322cce572f98 2023-04-05 04:58:53,954 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/4f27228c411a4f1b972624702521872e as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/4f27228c411a4f1b972624702521872e 2023-04-05 04:58:53,957 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/4f27228c411a4f1b972624702521872e, entries=150, sequenceid=60, filesize=25.1 K 2023-04-05 04:58:53,957 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/ff9381ebcd8749cd8e07ab5d55e75da9 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/ff9381ebcd8749cd8e07ab5d55e75da9 2023-04-05 04:58:53,963 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/ff9381ebcd8749cd8e07ab5d55e75da9, entries=150, sequenceid=60, filesize=11.6 K 2023-04-05 04:58:53,963 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/701b0d86095b4c94ae1b322cce572f98 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/701b0d86095b4c94ae1b322cce572f98 2023-04-05 04:58:53,968 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/701b0d86095b4c94ae1b322cce572f98, entries=150, sequenceid=60, filesize=11.6 K 2023-04-05 04:58:53,969 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=136.42 KB/139690 for ae76f0bbec3b73d99b1c36f6bc76ae4f in 1249ms, sequenceid=60, compaction requested=true 2023-04-05 04:58:53,969 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:58:53,969 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:53,969 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:53,969 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:53,969 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:53,969 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:53,969 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:53,970 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:53,970 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:53,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:53,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:53,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:53,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:53,971 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:53,971 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:53,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:53,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:53,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:53,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:53,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:53,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41124b1e[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:53,972 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:53,972 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:53,972 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:53,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:53,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:53,973 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:53,973 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:53,973 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:53,973 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:53,973 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:53,973 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:53,973 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:53,973 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:53,973 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:53,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:53,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:53,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:53,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:53,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:53,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:53,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:53,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:53,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:53,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:53,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:53,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:53,976 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:53,976 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:53,976 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:53,976 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:53,976 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:53,977 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:53,977 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:53,977 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:53,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:53,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:53,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:53,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:53,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:53,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:53,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:53,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:53,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:53,982 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:53,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:53,983 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:53,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:53,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:53,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:58:53,988 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ae76f0bbec3b73d99b1c36f6bc76ae4f 3/3 column families, dataSize=143.13 KB heapSize=410.69 KB 2023-04-05 04:58:53,988 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=A 2023-04-05 04:58:53,989 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-05 04:58:53,989 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=B 2023-04-05 04:58:53,989 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-05 04:58:53,989 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=C 2023-04-05 04:58:53,989 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-05 04:58:53,992 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:53,992 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:53,992 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:53,992 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:53,992 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:53,992 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:53,992 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:53,992 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:53,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:53,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:53,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:53,992 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:53,993 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:53,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:53,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:53,993 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:53,993 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:53,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:53,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:54,012 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:54,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38124 deadline: 1680670794010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,019 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:54,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38064 deadline: 1680670794018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,022 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:54,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38120 deadline: 1680670794020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,022 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:54,022 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:54,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38086 deadline: 1680670794021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38100 deadline: 1680670794020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,093 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:54,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:54,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:54,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:54,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:54,096 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:54,096 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:54,096 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:54,096 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:54,096 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,096 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,097 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:54,097 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:54,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:54,097 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,097 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:54,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:54,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,098 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:54,098 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:54,098 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:54,098 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:54,098 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:54,098 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:54,098 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:54,098 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:54,098 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:54,098 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:58:54,098 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:58:54,098 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:54,098 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:54,098 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:54,098 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,098 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:54,098 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:54,099 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:54,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:54,099 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:54,100 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,100 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:54,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:54,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:54,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:54,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@348a3535[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:54,101 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:54,101 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,101 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,101 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:54,101 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:54,101 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:54,102 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,102 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:54,102 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,102 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:54,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:54,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:54,102 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:54,102 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:54,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:54,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:54,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:54,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:54,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:54,105 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:54,105 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:54,105 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:54,105 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:54,105 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:54,105 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,105 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,105 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:54,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:54,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:54,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:54,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:54,106 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:54,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:54,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,107 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:54,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,109 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,109 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:54,109 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,109 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:54,109 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:54,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:54,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:54,109 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:54,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:54,109 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:54,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:54,109 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,109 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:54,109 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:54,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:54,109 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,109 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,115 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:54,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38124 deadline: 1680670794112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,209 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:54,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:54,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:54,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:54,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:54,212 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:54,212 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:54,212 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:54,212 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:54,212 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,212 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,213 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:54,213 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:54,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:54,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,214 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,214 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:54,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:54,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,214 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:54,215 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:54,215 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:54,215 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:54,215 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:54,215 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:54,215 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:54,215 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:54,215 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:54,215 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:58:54,215 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:58:54,215 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:54,215 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:54,215 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:54,215 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,215 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:54,215 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:54,216 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,216 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:54,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:54,216 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,216 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:54,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:54,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:54,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:54,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:54,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@741ac60b[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:54,218 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:54,218 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,218 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,218 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:54,218 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:54,218 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:54,218 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,218 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:54,219 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:54,219 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:54,219 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,219 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:54,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:54,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:54,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:54,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:54,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:54,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:54,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:54,221 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:54,221 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:54,221 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:54,221 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:54,221 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:54,222 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,222 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:54,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:54,222 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:54,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:54,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:54,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:54,222 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:54,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,223 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:54,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,225 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,225 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:54,225 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:54,225 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:54,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:54,225 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:54,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:54,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:54,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:54,225 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,225 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,225 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,225 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,225 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:54,225 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,225 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:54,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:54,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:54,317 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:54,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38124 deadline: 1680670794316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,325 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:54,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:54,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:54,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:54,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:54,328 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:54,328 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:54,328 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:54,328 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:54,328 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,328 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,329 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,329 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:54,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:54,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:54,329 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,329 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:54,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:54,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,330 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:54,330 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:54,330 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:54,330 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:54,330 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:54,330 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:54,331 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:54,331 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:54,331 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:54,331 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:58:54,331 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:58:54,331 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:54,331 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:54,331 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:54,331 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,331 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:54,331 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:54,332 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,332 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:54,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:54,332 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,332 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:54,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:54,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:54,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:54,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:54,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e6fd742[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:54,333 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:54,333 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,333 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,334 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:54,334 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,334 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:54,334 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:54,334 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:54,334 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:54,334 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,334 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:54,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:54,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:54,334 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:54,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:54,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:54,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:54,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:54,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:54,336 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:54,336 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:54,336 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:54,336 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:54,336 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:54,337 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,337 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,337 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:54,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:54,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:54,337 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:54,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:54,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:54,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:54,338 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:54,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,340 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:54,340 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:54,340 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:54,340 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:54,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:54,340 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,340 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:54,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:54,340 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:54,340 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:54,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:54,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:54,340 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:54,340 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,340 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,340 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,402 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405e1e6ccf0033e430e96a95c29131d92a0 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405e1e6ccf0033e430e96a95c29131d92a0 2023-04-05 04:58:54,440 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:54,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:54,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:54,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:54,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:54,443 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:54,443 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:54,443 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:54,443 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:54,444 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,444 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,444 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,444 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:54,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:54,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:54,445 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,445 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:54,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:54,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,445 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:54,445 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:54,446 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:54,446 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:54,446 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:54,446 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:54,446 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:54,446 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:54,446 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:54,446 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:58:54,446 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:58:54,446 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:54,446 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:54,446 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:54,446 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,446 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:54,446 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:54,447 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,447 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:54,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:54,447 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,447 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:54,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:54,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:54,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:54,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:54,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4df6352a[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:54,448 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:54,448 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,449 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,449 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:54,449 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:54,449 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:54,449 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,449 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:54,449 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:54,449 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:54,449 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,449 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:54,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:54,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:54,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:54,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:54,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:54,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:54,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:54,451 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:54,451 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:54,451 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:54,451 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:54,451 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:54,452 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,452 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:54,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:54,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,452 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:54,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:54,453 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:54,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,454 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:54,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,457 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,457 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:54,457 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:54,457 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,457 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:54,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:54,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:54,457 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,457 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:54,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:54,457 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,457 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:54,457 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,457 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:54,457 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:54,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:54,557 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:54,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:54,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:54,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:54,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:54,560 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:54,560 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:54,560 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:54,560 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:54,561 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,561 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,561 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:54,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:54,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:54,561 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,562 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,562 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:54,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:54,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,562 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:54,562 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:54,562 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:54,562 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:54,562 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:54,562 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:54,563 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:54,563 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:54,563 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:54,563 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:58:54,563 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:58:54,563 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:54,563 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:54,563 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:54,563 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,563 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:54,563 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:54,564 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:54,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:54,564 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:54,564 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,564 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:54,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:54,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:54,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:54,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28b0823c[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:54,565 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:54,565 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,565 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,566 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:54,566 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:54,566 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:54,566 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,566 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:54,566 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,566 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:54,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:54,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:54,566 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:54,566 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:54,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:54,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:54,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:54,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:54,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:54,568 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:54,568 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:54,568 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:54,568 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:54,568 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:54,569 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,569 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,569 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:54,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:54,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:54,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:54,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:54,569 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:54,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:54,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,570 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:54,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,571 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:54,571 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:54,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:54,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:54,571 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:54,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:54,571 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:54,571 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:54,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:54,571 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:54,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:54,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:54,573 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,573 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,573 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,573 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,573 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,573 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,573 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,620 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:54,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38124 deadline: 1680670794618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,672 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:54,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:54,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:54,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:54,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:54,674 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:54,675 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:54,675 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:54,675 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:54,675 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,675 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,676 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,676 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:54,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:54,676 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,676 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,676 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:54,677 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:54,677 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:54,677 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:54,677 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:54,677 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:54,677 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:54,677 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:54,677 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:54,677 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:58:54,677 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:58:54,677 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:54,677 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:54,677 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:54,677 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,677 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:54,677 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:54,678 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:54,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:54,678 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:54,679 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,679 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:54,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:54,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:54,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:54,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ab833bf[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:54,680 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:54,680 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,680 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,680 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,680 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:54,680 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:54,680 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:54,680 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:54,681 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,681 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:54,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:54,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:54,681 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:54,681 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:54,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:54,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:54,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:54,683 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:54,683 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:54,683 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:54,683 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:54,683 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:54,684 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:54,684 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,684 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:54,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:54,684 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:54,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,685 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:54,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,686 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:54,686 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,686 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:54,686 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:54,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:54,686 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:54,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:54,686 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:54,686 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:54,687 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:54,686 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,687 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:54,687 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:54,687 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,787 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:54,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:54,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:54,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:54,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:54,789 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:54,790 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:54,790 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:54,790 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:54,790 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,790 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,791 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,791 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:54,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:54,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:54,791 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,791 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:54,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,791 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:54,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,791 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:54,792 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:54,793 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:54,793 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:54,793 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:54,793 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:54,793 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:54,793 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:54,793 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:54,793 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:58:54,793 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:58:54,793 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:54,793 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:54,793 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:54,793 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,793 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:54,793 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:54,794 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:54,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:54,794 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:54,803 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,803 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:54,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:54,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:54,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:54,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a8d206d[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:54,804 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:54,804 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,806 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=98, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/c8d2beff817842d69852c73b72c36924 2023-04-05 04:58:54,808 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,809 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:54,809 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:54,809 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:54,809 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,809 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:54,810 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:54,810 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,810 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:54,810 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:54,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:54,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:54,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:54,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:54,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:54,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:54,815 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:54,815 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:54,815 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:54,815 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:54,816 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:54,817 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,817 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,817 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:54,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:54,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:54,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:54,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:54,818 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:54,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:54,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,819 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:54,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,820 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:54,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:54,820 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:54,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:54,820 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,820 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,820 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,820 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,820 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:54,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:54,820 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:54,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:54,820 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:54,821 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:54,821 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:54,821 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,921 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:54,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:54,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:54,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:54,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:54,923 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:54,923 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:54,924 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:54,924 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:54,924 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,924 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,925 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,925 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:54,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:54,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:54,926 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,926 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,926 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:54,927 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:54,927 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:54,927 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:54,927 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:54,927 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:54,927 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:54,927 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:54,927 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:54,927 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:58:54,927 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:58:54,927 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:54,927 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:54,927 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:54,927 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,927 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:54,927 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:54,928 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,928 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:54,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:54,928 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,928 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:54,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:54,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:54,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:54,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:54,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@162e0773[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:54,929 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:54,929 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,930 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,930 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:54,930 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,930 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:54,930 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:54,930 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:54,931 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:54,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:54,931 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:54,931 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:54,931 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:54,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:54,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:54,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:54,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:54,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:54,933 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:54,933 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:54,933 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:54,933 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:54,933 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:54,934 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,934 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,934 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:54,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:54,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:54,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,934 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:54,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:54,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:54,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:54,935 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:54,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:54,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,937 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,937 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:54,937 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,937 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:54,937 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:54,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:54,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:54,937 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:54,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:54,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:54,937 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:54,937 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,937 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:54,937 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:54,937 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:54,937 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:54,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:54,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:55,025 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:55,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38064 deadline: 1680670795025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,029 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:55,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38086 deadline: 1680670795027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:55,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38100 deadline: 1680670795031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,033 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:55,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38120 deadline: 1680670795032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,037 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:55,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:55,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:55,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:55,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:55,040 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:55,040 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:55,040 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:55,040 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:55,040 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,040 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:55,041 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:55,041 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:55,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:55,041 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:55,041 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:55,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:55,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:55,042 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:55,042 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:55,042 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:55,042 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:55,042 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:55,042 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:55,042 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:55,042 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:55,042 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:55,042 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:58:55,042 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:58:55,042 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:55,042 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:55,042 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:55,042 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,042 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:55,043 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:55,043 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,043 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:55,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:55,044 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,044 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:55,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:55,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:55,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:55,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:55,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:55,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a23c6e8[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:55,045 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:55,045 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,045 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,045 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:55,045 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:55,045 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,045 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:55,045 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:55,046 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,046 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:55,046 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:55,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:55,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:55,046 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:55,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:55,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:55,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:55,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:55,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:55,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:55,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:55,048 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:55,048 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:55,048 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:55,048 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:55,048 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:55,049 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,049 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,049 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:55,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:55,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:55,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:55,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:55,049 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:55,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:55,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:55,050 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:55,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,052 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,052 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:55,052 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:55,052 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,052 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,052 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,052 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,052 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:55,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:55,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:55,052 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:55,052 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:55,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:55,054 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:55,054 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:55,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:55,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:55,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:55,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:55,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:55,124 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:55,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38124 deadline: 1680670795123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,152 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:55,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:55,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:55,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:55,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:55,155 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:55,155 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:55,155 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:55,155 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:55,155 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,155 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:55,156 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,156 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:55,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:55,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:55,156 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,156 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:55,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:55,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:55,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:55,157 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:55,157 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:55,157 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:55,157 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:55,157 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:55,157 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:55,157 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:55,157 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:55,157 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:55,157 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:58:55,157 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:58:55,157 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:55,157 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:55,157 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:55,158 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,158 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:55,158 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:55,158 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,158 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:55,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:55,159 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,159 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:55,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:55,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:55,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:55,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:55,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:55,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71cd3fd8[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:55,160 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:55,161 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,161 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,161 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:55,161 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:55,161 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,161 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:55,161 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:55,162 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,162 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:55,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:55,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:55,162 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:55,162 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:55,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:55,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:55,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:55,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:55,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:55,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:55,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:55,164 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:55,164 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:55,164 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:55,164 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:55,164 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:55,165 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,165 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:55,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:55,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,165 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:55,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:55,165 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:55,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:55,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:55,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:55,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:55,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,166 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:55,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:55,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,167 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:55,167 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:55,168 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:55,168 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:55,167 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:55,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:55,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:55,168 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:55,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:55,168 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,168 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,168 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,168 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,168 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:55,168 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:55,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:55,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:55,218 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/dc85a1c74c524df1be40a21610aba247 2023-04-05 04:58:55,268 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:55,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:55,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:55,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:55,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:55,271 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:55,271 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:55,271 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:55,271 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:55,271 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,271 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:55,272 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,272 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:55,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:55,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:55,272 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,272 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:55,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:55,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:55,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:55,273 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:55,273 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:55,273 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:55,273 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:55,273 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:55,273 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:55,273 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:55,273 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:55,273 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:55,273 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:58:55,273 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:58:55,273 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:55,273 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:55,273 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:55,273 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,273 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:55,274 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:55,274 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,274 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:55,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:55,274 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,274 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:55,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:55,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:55,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:55,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:55,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:55,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72c0f7c4[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:55,276 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:55,276 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,277 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,277 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:55,277 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:55,277 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:55,278 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,278 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:55,278 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:55,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:55,278 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:55,278 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:55,278 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:55,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:55,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:55,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:55,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:55,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:55,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:55,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:55,280 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:55,280 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:55,280 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:55,280 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:55,280 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:55,281 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,281 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:55,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,281 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:55,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:55,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:55,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:55,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:55,282 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:55,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:55,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:55,283 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:55,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:55,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,284 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:55,284 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,284 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:55,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:55,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:55,284 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:55,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:55,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:55,284 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:55,284 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:55,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:55,285 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,285 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,285 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,285 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,285 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:55,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:55,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:55,285 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:55,385 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:55,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:55,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:55,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:55,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:55,388 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:55,388 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:55,388 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:55,388 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:55,388 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,388 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:55,389 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:55,389 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:55,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:55,389 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,389 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:55,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:55,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:55,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:55,390 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:55,390 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:55,390 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:55,390 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:55,390 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:55,390 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:55,390 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:55,390 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:55,390 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:55,390 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:58:55,390 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:58:55,390 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:55,390 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:55,390 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:55,390 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,390 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:55,390 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:55,391 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:55,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:55,391 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:55,392 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,392 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:55,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:55,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:55,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:55,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:55,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14b36370[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:55,393 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:55,393 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,393 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,393 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:55,393 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:55,393 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:55,393 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,393 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:55,394 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,394 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:55,394 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:55,394 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:55,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:55,396 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:55,396 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:55,396 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:55,396 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:55,396 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:55,396 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,396 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:55,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,396 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:55,397 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:55,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,398 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:55,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:55,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,399 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,399 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:55,399 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:55,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:55,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:55,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:55,399 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,399 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:55,399 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:55,399 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:55,399 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:55,399 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:55,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:55,399 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:55,400 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,400 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:55,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:55,499 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:55,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:55,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:55,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:55,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:55,502 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:55,502 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:55,502 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:55,502 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:55,503 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,503 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:55,503 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:55,503 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:55,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:55,504 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,504 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:55,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:55,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:55,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:55,504 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:55,504 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:55,504 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:55,504 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:55,504 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:55,504 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:55,504 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:55,504 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:55,504 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:55,505 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:58:55,505 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:58:55,505 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:55,505 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:55,505 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:55,505 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,505 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:55,505 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:55,505 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,505 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:55,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:55,506 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,506 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:55,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:55,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:55,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:55,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:55,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:55,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3003e388[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:55,507 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:55,507 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,507 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,507 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:55,507 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:55,507 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:55,507 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,507 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:55,508 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:55,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:55,508 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:55,508 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:55,508 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:55,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:55,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:55,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:55,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:55,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:55,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:55,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:55,510 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:55,510 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:55,510 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:55,510 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:55,510 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:55,511 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,511 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,511 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:55,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:55,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:55,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,511 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:55,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:55,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:55,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:55,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:55,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:55,512 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:55,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:55,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,513 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,513 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:55,513 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:55,513 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:55,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:55,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:55,513 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:55,513 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:55,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:55,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:55,513 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,513 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:55,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:55,513 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:55,513 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,513 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,513 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:55,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:55,613 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:55,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:55,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:55,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:55,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:55,616 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:55,616 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:55,616 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:55,616 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:55,616 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,616 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:55,617 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:55,617 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:55,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:55,617 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,617 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:55,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:55,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:55,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:55,618 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:55,618 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:55,618 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:55,618 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:55,618 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:55,618 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:55,618 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:55,618 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:55,618 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:55,618 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:58:55,618 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:58:55,618 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:55,618 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:55,618 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:55,618 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,618 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:55,619 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:55,619 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:55,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:55,619 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:55,620 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,620 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:55,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:55,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:55,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:55,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:55,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@282829af[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:55,621 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:55,621 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,621 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,621 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:55,621 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:55,621 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:55,621 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,622 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:55,622 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,622 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:55,622 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:55,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:55,622 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:55,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:55,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:55,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:55,624 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:55,624 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:55,624 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:55,624 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:55,624 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:55,625 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,625 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:55,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:55,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,625 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:55,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:55,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:55,626 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:55,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:55,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:55,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:55,627 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:55,627 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/f517064e4a4a43f68bd5f19c1c1ccea9 2023-04-05 04:58:55,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:55,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,629 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,629 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:55,629 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:55,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:55,629 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:55,629 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:55,629 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:55,629 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,629 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,629 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:55,629 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:55,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:55,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:55,629 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:55,629 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:55,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:55,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:55,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:55,630 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/c8d2beff817842d69852c73b72c36924 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/c8d2beff817842d69852c73b72c36924 2023-04-05 04:58:55,634 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/c8d2beff817842d69852c73b72c36924, entries=150, sequenceid=98, filesize=25.1 K 2023-04-05 04:58:55,634 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/dc85a1c74c524df1be40a21610aba247 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/dc85a1c74c524df1be40a21610aba247 2023-04-05 04:58:55,637 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/dc85a1c74c524df1be40a21610aba247, entries=150, sequenceid=98, filesize=11.6 K 2023-04-05 04:58:55,637 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/f517064e4a4a43f68bd5f19c1c1ccea9 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/f517064e4a4a43f68bd5f19c1c1ccea9 2023-04-05 04:58:55,640 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/f517064e4a4a43f68bd5f19c1c1ccea9, entries=150, sequenceid=98, filesize=11.6 K 2023-04-05 04:58:55,640 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~143.13 KB/146560, heapSize ~410.64 KB/420496, currentSize=40.25 KB/41220 for ae76f0bbec3b73d99b1c36f6bc76ae4f in 1652ms, sequenceid=98, compaction requested=true 2023-04-05 04:58:55,641 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:58:55,641 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:58:55,641 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:58:55,641 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:58:55,641 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:58:55,641 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:58:55,642 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:58:55,642 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): ae76f0bbec3b73d99b1c36f6bc76ae4f/B is initiating minor compaction (all files) 2023-04-05 04:58:55,642 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 96036 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:58:55,642 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ae76f0bbec3b73d99b1c36f6bc76ae4f/B in TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:55,642 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): ae76f0bbec3b73d99b1c36f6bc76ae4f/A is initiating minor compaction (all files) 2023-04-05 04:58:55,642 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ae76f0bbec3b73d99b1c36f6bc76ae4f/A in TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:55,642 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/c9b802fd70b14ef5b9ff33b9fe31ebfb, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/62c669d8076a43ab831e658745683c4b, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/ff9381ebcd8749cd8e07ab5d55e75da9, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/dc85a1c74c524df1be40a21610aba247] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp, totalSize=44.1 K 2023-04-05 04:58:55,642 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/d5ec479c08424a7b9783c4833718aaf7, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/1c9cc481f75f4df48f7ad6a75cd2fc0c, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/4f27228c411a4f1b972624702521872e, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/c8d2beff817842d69852c73b72c36924] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp, totalSize=93.8 K 2023-04-05 04:58:55,642 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting c9b802fd70b14ef5b9ff33b9fe31ebfb, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680670730717 2023-04-05 04:58:55,643 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting d5ec479c08424a7b9783c4833718aaf7, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680670730717 2023-04-05 04:58:55,643 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 62c669d8076a43ab831e658745683c4b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680670730753 2023-04-05 04:58:55,643 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 1c9cc481f75f4df48f7ad6a75cd2fc0c, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680670730753 2023-04-05 04:58:55,643 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting ff9381ebcd8749cd8e07ab5d55e75da9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680670731857 2023-04-05 04:58:55,643 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 4f27228c411a4f1b972624702521872e, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680670731857 2023-04-05 04:58:55,643 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting dc85a1c74c524df1be40a21610aba247, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680670732907 2023-04-05 04:58:55,643 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting c8d2beff817842d69852c73b72c36924, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680670732907 2023-04-05 04:58:55,648 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): ae76f0bbec3b73d99b1c36f6bc76ae4f#B#compaction#470 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:58:55,652 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): ae76f0bbec3b73d99b1c36f6bc76ae4f#A#compaction#471 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:58:55,729 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:55,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:55,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:55,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:55,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:55,732 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:55,732 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:55,732 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:55,732 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:55,732 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,732 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:55,733 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,733 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:55,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:55,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:55,733 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:55,733 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:55,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:55,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:55,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:55,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:55,733 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:55,734 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:55,734 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:55,734 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:55,734 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:55,734 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:55,734 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:55,734 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:55,734 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:58:55,734 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ae76f0bbec3b73d99b1c36f6bc76ae4f 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-05 04:58:55,734 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=A 2023-04-05 04:58:55,734 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:55,735 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=B 2023-04-05 04:58:55,735 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:55,735 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=C 2023-04-05 04:58:55,735 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:55,744 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304052e9493d1b76845ce9820b3b6ec859c4d to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304052e9493d1b76845ce9820b3b6ec859c4d 2023-04-05 04:58:56,055 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/33107c9dcd8f4354b5db6916d9f0e883 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/33107c9dcd8f4354b5db6916d9f0e883 2023-04-05 04:58:56,059 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ae76f0bbec3b73d99b1c36f6bc76ae4f/B of ae76f0bbec3b73d99b1c36f6bc76ae4f into 33107c9dcd8f4354b5db6916d9f0e883(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:58:56,059 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:58:56,059 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f., storeName=ae76f0bbec3b73d99b1c36f6bc76ae4f/B, priority=12, startTime=1680670735641; duration=0sec 2023-04-05 04:58:56,059 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:58:56,059 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:58:56,062 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:58:56,063 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): ae76f0bbec3b73d99b1c36f6bc76ae4f/C is initiating minor compaction (all files) 2023-04-05 04:58:56,063 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ae76f0bbec3b73d99b1c36f6bc76ae4f/C in TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:56,063 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/c32fca2ae9eb49a99a7cc2e9d251ebbd, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/6c45d31e0a9040189de6c478d88c3448, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/701b0d86095b4c94ae1b322cce572f98, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/f517064e4a4a43f68bd5f19c1c1ccea9] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp, totalSize=44.1 K 2023-04-05 04:58:56,063 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting c32fca2ae9eb49a99a7cc2e9d251ebbd, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680670730717 2023-04-05 04:58:56,063 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 6c45d31e0a9040189de6c478d88c3448, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680670730753 2023-04-05 04:58:56,064 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 701b0d86095b4c94ae1b322cce572f98, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680670731857 2023-04-05 04:58:56,064 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting f517064e4a4a43f68bd5f19c1c1ccea9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680670732907 2023-04-05 04:58:56,073 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): ae76f0bbec3b73d99b1c36f6bc76ae4f#C#compaction#473 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:58:56,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:58:56,131 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:58:56,150 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=107, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/3740d85fee78431a96885921cc09bfee 2023-04-05 04:58:56,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670736174, max timestamp=1680670736183Num uniques 859; 2023-04-05 04:58:56,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:58:56,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670736174, max timestamp=1680670736183Num uniques 859; 2023-04-05 04:58:56,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:58:56,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680670736174, max timestamp=1680670736183Num uniques 859; 2023-04-05 04:58:56,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-05 04:58:56,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680670736222, max timestamp=1680670736234Num uniques 1018; 2023-04-05 04:58:56,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-05 04:58:56,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680670736222, max timestamp=1680670736234Num uniques 1018; 2023-04-05 04:58:56,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-05 04:58:56,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680670736222, max timestamp=1680670736234Num uniques 1009; 2023-04-05 04:58:56,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-04-05 04:58:56,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680670736273, max timestamp=1680670736288Num uniques 1027; 2023-04-05 04:58:56,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680670736273, max timestamp=1680670736288Num uniques 1027; 2023-04-05 04:58:56,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-05 04:58:56,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680670736273, max timestamp=1680670736288Num uniques 1027; 2023-04-05 04:58:56,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-05 04:58:56,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-05 04:58:56,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680670736326, max timestamp=1680670736340Num uniques 1036; 2023-04-05 04:58:56,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680670736326, max timestamp=1680670736340Num uniques 1036; 2023-04-05 04:58:56,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-05 04:58:56,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-05 04:58:56,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680670736326, max timestamp=1680670736340Num uniques 1036; 2023-04-05 04:58:56,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-05 04:58:56,403 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680670736392, max timestamp=1680670736400Num uniques 1045; 2023-04-05 04:58:56,403 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-05 04:58:56,403 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680670736392, max timestamp=1680670736400Num uniques 1045; 2023-04-05 04:58:56,403 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680670736392, max timestamp=1680670736400Num uniques 1045; 2023-04-05 04:58:56,403 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-05 04:58:56,403 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-05 04:58:56,458 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:56,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38124 deadline: 1680670796455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:56,473 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/d54123f6f469412d92b670130d3f308e as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/d54123f6f469412d92b670130d3f308e 2023-04-05 04:58:56,476 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ae76f0bbec3b73d99b1c36f6bc76ae4f/A of ae76f0bbec3b73d99b1c36f6bc76ae4f into d54123f6f469412d92b670130d3f308e(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:58:56,476 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:58:56,476 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f., storeName=ae76f0bbec3b73d99b1c36f6bc76ae4f/A, priority=12, startTime=1680670735641; duration=0sec 2023-04-05 04:58:56,476 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:58:56,479 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/14c954960e6140adb6f1c7b94c0f9924 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/14c954960e6140adb6f1c7b94c0f9924 2023-04-05 04:58:56,482 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ae76f0bbec3b73d99b1c36f6bc76ae4f/C of ae76f0bbec3b73d99b1c36f6bc76ae4f into 14c954960e6140adb6f1c7b94c0f9924(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:58:56,482 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:58:56,482 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f., storeName=ae76f0bbec3b73d99b1c36f6bc76ae4f/C, priority=12, startTime=1680670735641; duration=0sec 2023-04-05 04:58:56,482 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:58:56,557 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/56c1eb9505cd4d35b745bb80b6f1e5a2 2023-04-05 04:58:56,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:56,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38124 deadline: 1680670796559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:56,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:56,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38124 deadline: 1680670796762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:56,969 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/86d51637b14743999dd9800e3792830a 2023-04-05 04:58:56,972 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/3740d85fee78431a96885921cc09bfee as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/3740d85fee78431a96885921cc09bfee 2023-04-05 04:58:56,976 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/3740d85fee78431a96885921cc09bfee, entries=150, sequenceid=107, filesize=25.1 K 2023-04-05 04:58:56,976 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/56c1eb9505cd4d35b745bb80b6f1e5a2 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/56c1eb9505cd4d35b745bb80b6f1e5a2 2023-04-05 04:58:56,979 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/56c1eb9505cd4d35b745bb80b6f1e5a2, entries=150, sequenceid=107, filesize=11.6 K 2023-04-05 04:58:56,980 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/86d51637b14743999dd9800e3792830a as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/86d51637b14743999dd9800e3792830a 2023-04-05 04:58:56,982 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/86d51637b14743999dd9800e3792830a, entries=150, sequenceid=107, filesize=11.6 K 2023-04-05 04:58:56,983 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for ae76f0bbec3b73d99b1c36f6bc76ae4f in 1249ms, sequenceid=107, compaction requested=false 2023-04-05 04:58:56,983 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:58:56,983 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:56,983 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:56,983 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:56,983 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:56,983 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:56,983 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:56,984 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:56,984 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:56,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:56,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:56,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:56,984 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:56,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:56,984 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:56,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:56,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:56,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:56,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:56,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:56,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5da385b9[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:56,985 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:56,985 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:56,986 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:56,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:56,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:56,986 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:56,986 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:56,986 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:56,986 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:56,986 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:56,987 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:56,987 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:56,987 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:56,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:56,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:56,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:56,987 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:56,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:56,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:56,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:56,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:56,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:56,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:56,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:56,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:56,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:56,989 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:56,989 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:56,989 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:56,989 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:56,989 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:56,990 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:56,990 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:56,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:56,990 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:56,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:56,990 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:56,991 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:56,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:56,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:56,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:56,993 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:56,993 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:56,993 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:56,993 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:56,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:56,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:56,993 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:56,993 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:56,993 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:56,993 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:56,994 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:56,994 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:56,994 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:56,994 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:56,994 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:56,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:56,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:56,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:56,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:57,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:58:57,036 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ae76f0bbec3b73d99b1c36f6bc76ae4f 3/3 column families, dataSize=147.60 KB heapSize=422.16 KB 2023-04-05 04:58:57,036 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=A 2023-04-05 04:58:57,036 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-05 04:58:57,036 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=B 2023-04-05 04:58:57,036 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-05 04:58:57,036 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=C 2023-04-05 04:58:57,037 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-05 04:58:57,044 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:57,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38086 deadline: 1680670797041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,045 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:57,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38100 deadline: 1680670797042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,045 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:57,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38120 deadline: 1680670797043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,046 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:57,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38064 deadline: 1680670797044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,047 DEBUG [Thread-3535] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4138 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f., hostname=jenkins-hbase20.apache.org,38091,1680670569206, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 04:58:57,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:57,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38124 deadline: 1680670797065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,094 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:57,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:57,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:57,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:57,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:57,096 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:57,096 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:57,096 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:57,096 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:57,096 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,096 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,097 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:57,097 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:57,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:57,097 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,097 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:57,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,098 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:57,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,098 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:57,098 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:57,098 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:57,098 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:57,098 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:57,098 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:57,098 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:57,098 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:57,099 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:57,099 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:58:57,099 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:58:57,099 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:57,099 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:57,099 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:57,099 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,099 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:57,099 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:57,100 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,100 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:57,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:57,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:57,100 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,100 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:57,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:57,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:57,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:57,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5204d59e[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:57,101 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:57,101 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,102 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,102 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:57,102 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:57,102 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:57,102 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,102 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:57,102 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:57,102 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:57,102 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,102 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:57,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:57,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:57,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:57,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:57,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:57,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:57,104 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:57,104 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:57,104 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:57,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:57,104 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:57,104 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:57,105 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,105 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:57,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:57,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,105 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:57,106 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:57,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:57,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:57,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:57,107 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:57,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,109 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,109 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,109 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,109 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,109 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:57,109 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:57,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:57,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:57,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:57,109 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:57,109 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,110 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,110 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,110 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:57,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:57,110 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:57,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:57,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:57,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:57,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38086 deadline: 1680670797145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,149 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:57,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38100 deadline: 1680670797146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:57,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38064 deadline: 1680670797147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,210 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:57,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:57,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:57,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:57,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:57,212 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:57,212 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:57,212 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:57,212 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:57,213 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,213 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,213 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,213 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:57,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:57,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:57,214 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,214 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:57,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:57,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,214 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:57,214 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:57,214 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:57,214 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:57,214 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:57,214 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:57,215 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:57,215 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:57,215 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:57,215 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:58:57,215 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:58:57,215 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:57,215 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:57,215 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:57,215 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,215 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:57,215 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:57,216 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,216 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:57,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:57,216 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,216 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:57,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:57,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:57,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:57,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:57,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d3473f5[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:57,217 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:57,217 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,218 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,218 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:57,218 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:57,218 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,218 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:57,218 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:57,219 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,219 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:57,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:57,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:57,219 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:57,219 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:57,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:57,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:57,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:57,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:57,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:57,220 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:57,220 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:57,220 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:57,220 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:57,220 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:57,221 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,221 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:57,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,221 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:57,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:57,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:57,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:57,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:57,222 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:57,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,223 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:57,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,225 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,225 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,225 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,225 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,225 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,225 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,225 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:57,225 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:57,226 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:57,226 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:57,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:57,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:57,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:57,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:57,226 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:57,226 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:57,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:57,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:57,326 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:57,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:57,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:57,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:57,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:57,330 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:57,330 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:57,330 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:57,330 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:57,330 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,330 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,331 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:57,331 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:57,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,332 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,332 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:57,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,332 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:57,332 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:57,332 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:57,332 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:57,332 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:57,333 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:57,333 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:57,333 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:57,333 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:57,333 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:58:57,333 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:58:57,333 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:57,333 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:57,333 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:57,333 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,333 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:57,333 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:57,334 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,334 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:57,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:57,334 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,334 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:57,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:57,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:57,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:57,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:57,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@143858ad[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:57,335 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:57,335 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,336 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,336 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:57,336 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,336 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:57,336 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:57,336 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:57,337 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,337 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:57,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:57,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:57,337 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:57,337 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:57,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:57,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:57,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:57,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:57,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:57,349 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:57,349 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:57,349 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:57,349 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:57,349 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:57,350 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:57,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38086 deadline: 1680670797348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,352 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:57,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38100 deadline: 1680670797349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,353 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:57,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38064 deadline: 1680670797352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,358 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,358 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,358 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:57,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:57,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:57,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:57,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:57,359 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:57,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:57,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,360 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:57,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,362 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,362 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,362 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:57,362 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:57,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:57,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:57,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:57,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:57,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:57,362 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,362 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:57,362 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,363 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:57,363 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:57,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:57,363 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,363 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:57,363 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,446 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040539c092314046487a825b8e81ec16668d to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040539c092314046487a825b8e81ec16668d 2023-04-05 04:58:57,463 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:57,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:57,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:57,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:57,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:57,465 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:57,466 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:57,466 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:57,466 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:57,466 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,466 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,466 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,466 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:57,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:57,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:57,467 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,467 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:57,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:57,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,467 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:57,467 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:57,467 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:57,467 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:57,468 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:57,468 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:57,468 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:57,468 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:57,468 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:58:57,468 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:58:57,468 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:57,468 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:57,468 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:57,468 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,468 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:57,468 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:57,469 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,469 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:57,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:57,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:57,469 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,469 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:57,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:57,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:57,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:57,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@557789de[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:57,470 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:57,470 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,470 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,471 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:57,471 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:57,471 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,471 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:57,471 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:57,471 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,471 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:57,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:57,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:57,471 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:57,471 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:57,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:57,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:57,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:57,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:57,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:57,473 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:57,473 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:57,473 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:57,473 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:57,473 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:57,474 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,474 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:57,474 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:57,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:57,474 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:57,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,475 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:57,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,477 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:57,477 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:57,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:57,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:57,477 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,477 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,477 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,477 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:57,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:57,477 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,477 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:57,477 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:57,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:57,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:57,477 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,477 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:57,478 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:57,568 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:57,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38124 deadline: 1680670797567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,578 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:57,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:57,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:57,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:57,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:57,580 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:57,580 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:57,580 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:57,580 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:57,580 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,580 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,581 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:57,581 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:57,581 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,581 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,582 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:57,582 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:57,582 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:57,582 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:57,582 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:57,582 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:57,582 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:57,582 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:57,582 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:57,582 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:58:57,582 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:58:57,583 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:57,583 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:57,583 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:57,583 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,583 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:57,583 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:57,583 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,583 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:57,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:57,584 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,584 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:57,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:57,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:57,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:57,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:57,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@212b0ecb[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:57,585 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:57,585 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,585 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,586 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:57,586 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:57,586 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:57,586 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,586 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:57,586 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:57,586 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:57,586 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,586 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:57,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:57,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:57,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:57,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:57,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:57,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:57,588 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:57,588 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:57,588 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:57,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:57,588 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:57,588 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:57,590 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,590 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,590 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:57,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:57,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:57,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:57,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:57,590 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:57,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:57,591 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:57,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,593 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,593 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,593 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,593 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:57,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:57,593 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:57,593 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:57,593 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,593 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:57,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:57,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:57,593 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,593 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:57,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:57,593 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:57,593 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:57,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:57,653 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:57,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38086 deadline: 1680670797651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,656 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:57,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38100 deadline: 1680670797654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,656 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:57,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38064 deadline: 1680670797656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,693 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:57,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:57,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:57,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:57,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:57,696 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:57,696 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:57,696 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:57,696 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:57,697 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,697 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,697 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:57,697 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:57,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:57,698 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,698 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:57,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,699 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:57,699 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:57,699 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:57,699 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:57,699 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:57,699 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:57,699 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:57,699 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:57,699 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:57,699 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:58:57,699 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:58:57,699 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:57,699 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:57,699 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:57,699 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,699 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:57,699 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:57,700 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,700 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:57,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:57,700 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,700 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:57,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:57,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:57,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:57,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:57,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fde6ac0[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:57,701 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:57,702 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,702 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,702 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:57,702 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:57,702 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:57,702 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,702 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:57,703 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:57,703 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:57,703 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:57,703 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:57,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:57,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:57,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:57,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:57,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:57,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:57,705 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:57,705 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:57,705 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:57,705 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:57,705 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:57,706 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,706 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:57,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:57,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,706 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:57,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:57,706 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:57,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:57,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:57,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,708 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:57,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,709 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,709 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,709 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:57,709 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:57,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:57,709 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:57,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:57,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:57,709 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:57,709 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,710 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,710 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,710 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,710 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:57,710 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:57,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:57,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:57,810 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:57,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:57,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:57,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:57,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:57,812 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:57,812 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:57,812 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:57,812 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:57,813 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,813 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,813 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:57,813 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:57,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:57,813 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,814 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:57,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:57,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,814 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:57,814 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:57,814 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:57,814 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:57,814 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:57,814 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:57,814 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:57,814 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:57,814 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:57,814 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:58:57,814 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:58:57,815 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:57,815 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:57,815 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:57,815 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,815 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:57,815 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:57,815 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,815 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:57,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:57,816 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,816 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:57,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:57,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2975b8a6[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:57,817 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:57,817 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,817 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,817 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:57,817 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:57,817 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:57,817 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,817 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:57,818 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,818 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:57,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:57,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:57,818 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:57,818 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:57,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:57,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:57,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:57,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:57,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:57,819 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:57,820 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:57,820 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:57,820 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:57,820 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:57,820 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,820 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:57,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:57,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,820 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:57,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:57,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:57,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:57,821 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:57,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:57,822 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,823 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,823 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,823 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,823 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,823 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:57,823 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:57,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:57,823 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:57,823 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:57,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:57,824 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:57,824 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:57,824 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,824 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:57,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:57,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:57,850 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=217, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/76df34c5008643958b0b07d3f4a214ef 2023-04-05 04:58:57,924 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:57,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:57,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:57,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:57,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:57,927 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:57,927 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:57,927 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:57,927 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:57,927 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,927 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,928 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:57,928 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:57,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:57,928 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,928 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:57,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:57,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,929 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:57,929 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:57,929 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:57,929 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:57,929 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:57,929 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:57,929 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:57,929 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:57,929 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:57,929 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:58:57,929 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:58:57,929 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:57,929 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:57,929 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:57,929 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,929 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:57,929 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:57,930 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,930 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:57,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:57,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:57,931 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,931 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:57,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:57,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:57,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:57,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40a5ed2f[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:57,932 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:57,932 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,932 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,932 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:57,932 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:57,932 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,932 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:57,932 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:57,933 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:57,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:57,933 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:57,933 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:57,933 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:57,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:57,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:57,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:57,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:57,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:57,935 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:57,935 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:57,935 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:57,935 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:57,935 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:57,936 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,936 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:57,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,936 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:57,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:57,936 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:57,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:57,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:57,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:57,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:57,937 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:57,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:57,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,938 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,938 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:57,939 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:57,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:57,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:57,939 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:57,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:57,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:57,939 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:57,939 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:57,939 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:57,939 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:57,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:57,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:57,939 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:57,939 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,939 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:57,939 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,039 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:58,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:58,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:58,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:58,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:58,042 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:58,042 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:58,042 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:58,042 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:58,042 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,042 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:58,044 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:58,044 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:58,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:58,044 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,044 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:58,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:58,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:58,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:58,044 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:58,045 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:58,045 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:58,045 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:58,045 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:58,045 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:58,045 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:58,045 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:58,045 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:58,045 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:58:58,045 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:58:58,045 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:58,045 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:58,045 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:58,045 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,045 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:58,045 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:58,046 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,046 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:58,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:58,046 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,046 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:58,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:58,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:58,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:58,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:58,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:58,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3901eea6[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:58,047 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:58,048 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,048 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,048 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:58,048 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:58,048 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:58,048 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,048 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:58,049 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,049 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:58,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:58,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:58,049 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:58,049 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:58,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:58,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:58,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:58,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:58,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:58,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:58,051 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:58,051 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:58,051 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:58,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:58,051 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:58,051 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:58,052 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,052 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:58,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:58,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,052 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:58,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:58,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:58,052 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:58,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:58,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:58,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:58,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,053 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:58,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:58,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,056 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:58,056 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:58,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:58,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:58,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:58,056 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:58,057 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:58,056 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:58,057 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:58,056 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:58,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:58,057 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:58,057 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,057 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,057 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,057 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:58,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:58,157 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:58,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:58,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:58,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38086 deadline: 1680670798156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:58,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38064 deadline: 1680670798157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:58,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:58,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:58,160 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:58,160 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:58,160 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:58,160 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:58,160 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,160 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:58,161 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:58,161 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:58,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:58,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:58,162 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,162 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:58,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:58,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:58,162 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:58,162 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:58,162 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:58,162 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:58,162 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:58,163 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:58,163 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:58,163 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:58,163 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:58,163 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:58:58,163 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:58:58,163 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:58,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:58,163 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:58,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38100 deadline: 1680670798161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,163 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:58,163 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,163 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:58,163 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:58,164 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,164 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:58,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:58,164 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,164 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:58,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:58,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:58,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:58,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:58,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:58,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35851b2c[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:58,165 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:58,165 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,166 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,166 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:58,166 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,166 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:58,166 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:58,166 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:58,168 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,168 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:58,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:58,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:58,168 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:58,168 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:58,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:58,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:58,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:58,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:58,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:58,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:58,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:58,172 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:58,172 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:58,172 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:58,172 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:58,172 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:58,173 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,173 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:58,173 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:58,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:58,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:58,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:58,174 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:58,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:58,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:58,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:58,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,175 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:58,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:58,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,177 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,177 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:58,177 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:58,177 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:58,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:58,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:58,177 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:58,177 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,177 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:58,177 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,177 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:58,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:58,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:58,177 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:58,177 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,178 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:58,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:58,258 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/68005163ce124a04b7d9d2ba3f360cb0 2023-04-05 04:58:58,278 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:58,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:58,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:58,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:58,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:58,280 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:58,280 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:58,280 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:58,280 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:58,281 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,281 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:58,281 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,281 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:58,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:58,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:58,281 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,281 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:58,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:58,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:58,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:58,282 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:58,282 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:58,282 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:58,282 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:58,282 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:58,282 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:58,283 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:58,283 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:58,283 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:58,283 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:58:58,283 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:58:58,283 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:58,283 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:58,283 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:58,283 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,283 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:58,283 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:58,284 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,284 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:58,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:58,284 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,284 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:58,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:58,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:58,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:58,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:58,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:58,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@313135b2[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:58,285 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:58,285 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,286 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,286 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:58,286 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:58,286 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:58,286 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,286 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:58,286 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:58,286 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,287 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:58,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:58,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:58,287 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:58,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:58,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:58,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:58,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:58,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:58,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:58,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:58,288 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:58,288 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:58,288 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:58,288 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:58,288 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:58,289 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,289 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:58,289 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:58,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:58,290 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:58,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:58,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:58,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:58,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:58,290 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:58,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:58,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:58,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,292 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:58,292 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:58,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:58,292 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:58,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:58,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:58,292 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:58,292 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:58,292 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:58,292 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,292 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,292 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:58,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:58,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:58,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:58,292 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:58,292 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,292 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,392 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:58,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:58,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:58,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:58,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:58,395 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:58,395 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:58,395 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:58,395 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:58,395 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,395 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:58,396 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:58,396 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:58,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:58,396 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:58,396 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:58,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:58,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:58,397 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:58,397 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:58,397 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:58,397 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:58,397 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:58,397 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:58,397 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:58,397 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:58,397 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:58,397 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:58:58,397 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:58:58,397 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:58,397 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:58,397 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:58,397 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,398 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:58,398 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:58,399 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,399 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:58,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:58,399 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,399 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:58,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:58,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:58,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:58,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:58,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:58,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e927a2e[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:58,400 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:58,400 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,401 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,401 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:58,401 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:58,401 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:58,401 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,401 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:58,402 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:58,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:58,402 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:58,402 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:58,402 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:58,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:58,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:58,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:58,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:58,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:58,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:58,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:58,404 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:58,404 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:58,404 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:58,404 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:58,404 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:58,405 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:58,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:58,405 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,405 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:58,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:58,405 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:58,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:58,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:58,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:58,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:58,406 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:58,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:58,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,407 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:58,407 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,407 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:58,407 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:58,407 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:58,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:58,407 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:58,407 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:58,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:58,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:58,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:58,407 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:58,408 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:58,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:58,408 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,408 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:58,408 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,508 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:58,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:58,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:58,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:58,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:58,511 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:58,511 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:58,511 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:58,511 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:58,511 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,511 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:58,512 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:58,512 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:58,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:58,512 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,512 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:58,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:58,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:58,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:58,513 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:58,513 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:58,513 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:58,513 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:58,513 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:58,513 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:58,513 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:58,513 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:58,513 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:58,513 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:58:58,514 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:58:58,514 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:58,514 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:58,514 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:58,514 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,514 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:58,514 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:58,515 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,515 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:58,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:58,515 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,515 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:58,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:58,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:58,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:58,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:58,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:58,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6850cb84[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:58,516 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:58,516 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,516 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,517 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:58,517 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:58,517 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:58,517 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,517 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:58,517 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,517 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:58,518 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:58,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:58,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:58,518 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:58,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:58,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:58,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:58,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:58,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:58,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:58,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:58,519 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:58,519 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:58,520 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:58,520 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:58,520 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:58,520 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,520 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:58,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:58,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,520 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:58,521 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:58,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:58,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:58,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:58,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:58,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:58,521 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:58,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:58,524 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,524 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:58,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:58,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:58,524 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:58,525 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:58,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:58,524 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:58,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:58,524 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:58,525 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,525 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:58,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:58,525 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:58,525 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,525 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,525 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:58,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:58,577 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:58,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38124 deadline: 1680670798575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,625 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:58,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:58,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:58,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:58,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:58,628 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:58,628 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:58,628 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:58,628 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:58,628 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,628 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:58,629 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:58,629 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:58,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:58,629 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,629 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:58,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:58,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:58,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:58,629 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:58,630 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:58,630 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:58,630 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:58,630 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:58,630 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:58,630 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:58,630 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:58,630 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:58,630 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:58:58,630 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:58:58,630 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:58,630 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:58,630 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:58,630 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,630 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:58,630 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:58,631 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,631 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:58,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:58,631 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,631 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:58,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:58,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:58,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:58,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:58,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:58,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7591e1d9[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:58,632 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:58,632 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,633 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,633 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:58,633 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:58,633 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,633 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:58,633 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:58,634 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,634 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:58,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:58,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:58,634 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:58,634 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:58,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:58,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:58,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:58,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:58,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:58,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:58,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:58,635 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:58,635 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:58,635 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:58,635 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:58,635 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:58,636 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,636 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,636 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:58,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:58,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:58,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:58,637 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:58,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:58,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:58,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:58,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:58,638 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:58,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:58,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,639 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,639 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:58,639 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:58,639 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,639 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:58,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:58,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:58,639 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:58,639 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,639 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:58,639 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:58,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:58,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:58,639 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:58,639 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:58,639 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,640 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:58,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:58,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:58,668 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/4222d5bf2a014f9aada4ddf58b28d4ba 2023-04-05 04:58:58,674 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/76df34c5008643958b0b07d3f4a214ef as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/76df34c5008643958b0b07d3f4a214ef 2023-04-05 04:58:58,676 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/76df34c5008643958b0b07d3f4a214ef, entries=200, sequenceid=217, filesize=32.1 K 2023-04-05 04:58:58,677 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/68005163ce124a04b7d9d2ba3f360cb0 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/68005163ce124a04b7d9d2ba3f360cb0 2023-04-05 04:58:58,680 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/68005163ce124a04b7d9d2ba3f360cb0, entries=150, sequenceid=217, filesize=11.8 K 2023-04-05 04:58:58,681 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/4222d5bf2a014f9aada4ddf58b28d4ba as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/4222d5bf2a014f9aada4ddf58b28d4ba 2023-04-05 04:58:58,684 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/4222d5bf2a014f9aada4ddf58b28d4ba, entries=150, sequenceid=217, filesize=11.8 K 2023-04-05 04:58:58,684 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~422.11 KB/432240, currentSize=33.54 KB/34350 for ae76f0bbec3b73d99b1c36f6bc76ae4f in 1648ms, sequenceid=217, compaction requested=true 2023-04-05 04:58:58,684 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:58:58,684 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:58:58,684 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:58:58,684 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:58:58,684 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:58:58,684 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:58:58,689 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84516 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:58:58,689 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): ae76f0bbec3b73d99b1c36f6bc76ae4f/A is initiating minor compaction (all files) 2023-04-05 04:58:58,689 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35946 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:58:58,689 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ae76f0bbec3b73d99b1c36f6bc76ae4f/A in TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:58,689 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): ae76f0bbec3b73d99b1c36f6bc76ae4f/B is initiating minor compaction (all files) 2023-04-05 04:58:58,690 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ae76f0bbec3b73d99b1c36f6bc76ae4f/B in TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:58,690 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/d54123f6f469412d92b670130d3f308e, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/3740d85fee78431a96885921cc09bfee, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/76df34c5008643958b0b07d3f4a214ef] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp, totalSize=82.5 K 2023-04-05 04:58:58,690 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/33107c9dcd8f4354b5db6916d9f0e883, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/56c1eb9505cd4d35b745bb80b6f1e5a2, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/68005163ce124a04b7d9d2ba3f360cb0] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp, totalSize=35.1 K 2023-04-05 04:58:58,690 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting d54123f6f469412d92b670130d3f308e, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680670732907 2023-04-05 04:58:58,690 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 33107c9dcd8f4354b5db6916d9f0e883, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680670732907 2023-04-05 04:58:58,690 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 3740d85fee78431a96885921cc09bfee, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680670733995 2023-04-05 04:58:58,690 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 56c1eb9505cd4d35b745bb80b6f1e5a2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680670733995 2023-04-05 04:58:58,690 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 76df34c5008643958b0b07d3f4a214ef, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1680670736443 2023-04-05 04:58:58,690 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 68005163ce124a04b7d9d2ba3f360cb0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1680670736450 2023-04-05 04:58:58,695 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): ae76f0bbec3b73d99b1c36f6bc76ae4f#B#compaction#480 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:58:58,697 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): ae76f0bbec3b73d99b1c36f6bc76ae4f#A#compaction#479 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:58:58,731 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/3ca36fdff31f4b948c8436c5207a560c as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/3ca36fdff31f4b948c8436c5207a560c 2023-04-05 04:58:58,736 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ae76f0bbec3b73d99b1c36f6bc76ae4f/A of ae76f0bbec3b73d99b1c36f6bc76ae4f into 3ca36fdff31f4b948c8436c5207a560c(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:58:58,736 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:58:58,736 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f., storeName=ae76f0bbec3b73d99b1c36f6bc76ae4f/A, priority=13, startTime=1680670738684; duration=0sec 2023-04-05 04:58:58,736 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:58:58,736 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:58:58,737 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35946 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:58:58,737 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): ae76f0bbec3b73d99b1c36f6bc76ae4f/C is initiating minor compaction (all files) 2023-04-05 04:58:58,737 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ae76f0bbec3b73d99b1c36f6bc76ae4f/C in TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:58,737 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/14c954960e6140adb6f1c7b94c0f9924, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/86d51637b14743999dd9800e3792830a, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/4222d5bf2a014f9aada4ddf58b28d4ba] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp, totalSize=35.1 K 2023-04-05 04:58:58,738 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 14c954960e6140adb6f1c7b94c0f9924, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680670732907 2023-04-05 04:58:58,738 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 86d51637b14743999dd9800e3792830a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680670733995 2023-04-05 04:58:58,738 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 4222d5bf2a014f9aada4ddf58b28d4ba, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1680670736450 2023-04-05 04:58:58,740 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:58,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:58,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:58,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:58,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:58,758 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:58,758 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:58,758 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:58,758 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:58,758 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,758 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:58,759 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): ae76f0bbec3b73d99b1c36f6bc76ae4f#C#compaction#481 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:58:58,759 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,759 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:58,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:58,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:58,759 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:58,759 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:58,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:58,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:58,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:58,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:58,760 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:58,760 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:58,760 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:58,760 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:58,760 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:58,760 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:58,761 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:58,761 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:58,761 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:58,761 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:58:58,761 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ae76f0bbec3b73d99b1c36f6bc76ae4f 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-05 04:58:58,761 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=A 2023-04-05 04:58:58,761 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:58,761 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=B 2023-04-05 04:58:58,761 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:58,761 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=C 2023-04-05 04:58:58,761 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:58:59,113 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/8b6d1e6220cd433a9b51858476ab2c47 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/8b6d1e6220cd433a9b51858476ab2c47 2023-04-05 04:58:59,116 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ae76f0bbec3b73d99b1c36f6bc76ae4f/B of ae76f0bbec3b73d99b1c36f6bc76ae4f into 8b6d1e6220cd433a9b51858476ab2c47(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:58:59,116 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:58:59,116 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f., storeName=ae76f0bbec3b73d99b1c36f6bc76ae4f/B, priority=13, startTime=1680670738684; duration=0sec 2023-04-05 04:58:59,116 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:58:59,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:58:59,163 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:58:59,166 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/e513e2369c194d1babad81a3c9900670 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/e513e2369c194d1babad81a3c9900670 2023-04-05 04:58:59,175 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ae76f0bbec3b73d99b1c36f6bc76ae4f/C of ae76f0bbec3b73d99b1c36f6bc76ae4f into e513e2369c194d1babad81a3c9900670(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:58:59,175 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:58:59,175 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f., storeName=ae76f0bbec3b73d99b1c36f6bc76ae4f/C, priority=13, startTime=1680670738684; duration=0sec 2023-04-05 04:58:59,175 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:58:59,179 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040586e543e8dc73423bace00750f93bf35c to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040586e543e8dc73423bace00750f93bf35c 2023-04-05 04:58:59,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670739167, max timestamp=1680670739189Num uniques 859; 2023-04-05 04:58:59,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:58:59,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670739167, max timestamp=1680670739189Num uniques 859; 2023-04-05 04:58:59,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670739167, max timestamp=1680670739189Num uniques 859; 2023-04-05 04:58:59,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:58:59,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:58:59,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:59,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38086 deadline: 1680670799205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,208 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680670739198, max timestamp=1680670739206Num uniques 1068; 2023-04-05 04:58:59,208 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680670739198, max timestamp=1680670739206Num uniques 1068; 2023-04-05 04:58:59,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-05 04:58:59,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-05 04:58:59,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680670739198, max timestamp=1680670739206Num uniques 1068; 2023-04-05 04:58:59,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-05 04:58:59,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8092, totalHeapSize=18648, min timestamp=1680670739221, max timestamp=1680670739233Num uniques 1077; 2023-04-05 04:58:59,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=49312, new segment data size=8092, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-05 04:58:59,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8094, totalHeapSize=18648, min timestamp=1680670739221, max timestamp=1680670739233Num uniques 1077; 2023-04-05 04:58:59,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680670739221, max timestamp=1680670739233Num uniques 1077; 2023-04-05 04:58:59,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49314, new segment data size=8094, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-05 04:58:59,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-05 04:58:59,256 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680670739251, max timestamp=1680670739255Num uniques 1036; 2023-04-05 04:58:59,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680670739251, max timestamp=1680670739255Num uniques 1036; 2023-04-05 04:58:59,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-05 04:58:59,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-05 04:58:59,258 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680670739251, max timestamp=1680670739255Num uniques 1036; 2023-04-05 04:58:59,258 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-05 04:58:59,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680670739271, max timestamp=1680670739277Num uniques 1045; 2023-04-05 04:58:59,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-05 04:58:59,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680670739271, max timestamp=1680670739277Num uniques 1045; 2023-04-05 04:58:59,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680670739271, max timestamp=1680670739277Num uniques 1045; 2023-04-05 04:58:59,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-05 04:58:59,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-05 04:58:59,305 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680670739296, max timestamp=1680670739302Num uniques 1054; 2023-04-05 04:58:59,305 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-05 04:58:59,305 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680670739296, max timestamp=1680670739302Num uniques 1054; 2023-04-05 04:58:59,305 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-05 04:58:59,305 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680670739296, max timestamp=1680670739302Num uniques 1054; 2023-04-05 04:58:59,305 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-05 04:58:59,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680670739310, max timestamp=1680670739320Num uniques 1013; 2023-04-05 04:58:59,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:59,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680670739310, max timestamp=1680670739320Num uniques 1013; 2023-04-05 04:58:59,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38100 deadline: 1680670799319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-05 04:58:59,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-05 04:58:59,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680670739310, max timestamp=1680670739320Num uniques 1013; 2023-04-05 04:58:59,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-05 04:58:59,356 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:59,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38064 deadline: 1680670799355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,357 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680670739350, max timestamp=1680670739355Num uniques 1072; 2023-04-05 04:58:59,357 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-05 04:58:59,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680670739350, max timestamp=1680670739355Num uniques 1072; 2023-04-05 04:58:59,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-05 04:58:59,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680670739350, max timestamp=1680670739355Num uniques 1072; 2023-04-05 04:58:59,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-05 04:58:59,411 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680670739381, max timestamp=1680670739408Num uniques 1031; 2023-04-05 04:58:59,411 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680670739381, max timestamp=1680670739408Num uniques 1031; 2023-04-05 04:58:59,411 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680670739381, max timestamp=1680670739408Num uniques 1031; 2023-04-05 04:58:59,411 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-05 04:58:59,411 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-05 04:58:59,411 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-05 04:58:59,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680670739431, max timestamp=1680670739438Num uniques 1040; 2023-04-05 04:58:59,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-04-05 04:58:59,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680670739431, max timestamp=1680670739438Num uniques 1040; 2023-04-05 04:58:59,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-04-05 04:58:59,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680670739431, max timestamp=1680670739438Num uniques 1040; 2023-04-05 04:58:59,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-04-05 04:58:59,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9104, totalHeapSize=20936, min timestamp=1680670739460, max timestamp=1680670739465Num uniques 1049; 2023-04-05 04:58:59,467 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:59,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=48034, new segment data size=9104, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-05 04:58:59,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38086 deadline: 1680670799465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9104, totalHeapSize=20936, min timestamp=1680670739460, max timestamp=1680670739465Num uniques 1049; 2023-04-05 04:58:59,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=48034, new segment data size=9104, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-05 04:58:59,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9104, totalHeapSize=20936, min timestamp=1680670739460, max timestamp=1680670739465Num uniques 1049; 2023-04-05 04:58:59,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=48034, new segment data size=9104, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-05 04:58:59,499 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:59,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38064 deadline: 1680670799496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680670739479, max timestamp=1680670739498Num uniques 1058; 2023-04-05 04:58:59,500 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-05 04:58:59,500 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680670739479, max timestamp=1680670739498Num uniques 1058; 2023-04-05 04:58:59,500 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680670739479, max timestamp=1680670739498Num uniques 1058; 2023-04-05 04:58:59,500 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-05 04:58:59,500 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-05 04:58:59,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1680670739530, max timestamp=1680670739540Num uniques 1067; 2023-04-05 04:58:59,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-04-05 04:58:59,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1680670739530, max timestamp=1680670739540Num uniques 1067; 2023-04-05 04:58:59,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1680670739530, max timestamp=1680670739540Num uniques 1067; 2023-04-05 04:58:59,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-04-05 04:58:59,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-04-05 04:58:59,585 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=226, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/7a92851f34504a2495e92ba4ff7039fe 2023-04-05 04:58:59,594 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=226 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/335531c356a040a4ae376e7039d56e91 2023-04-05 04:58:59,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8046, totalHeapSize=18544, min timestamp=1680670739588, max timestamp=1680670739596Num uniques 1026; 2023-04-05 04:58:59,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46976, new segment data size=8046, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-04-05 04:58:59,599 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8049, totalHeapSize=18544, min timestamp=1680670739588, max timestamp=1680670739596Num uniques 1026; 2023-04-05 04:58:59,599 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46979, new segment data size=8049, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-04-05 04:58:59,599 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8046, totalHeapSize=18544, min timestamp=1680670739588, max timestamp=1680670739596Num uniques 1026; 2023-04-05 04:58:59,599 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46976, new segment data size=8046, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-04-05 04:58:59,603 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=226 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/700a9f652732416db63f544a0eb6408d 2023-04-05 04:58:59,606 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/7a92851f34504a2495e92ba4ff7039fe as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/7a92851f34504a2495e92ba4ff7039fe 2023-04-05 04:58:59,610 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/7a92851f34504a2495e92ba4ff7039fe, entries=150, sequenceid=226, filesize=25.3 K 2023-04-05 04:58:59,611 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/335531c356a040a4ae376e7039d56e91 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/335531c356a040a4ae376e7039d56e91 2023-04-05 04:58:59,616 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/335531c356a040a4ae376e7039d56e91, entries=150, sequenceid=226, filesize=11.8 K 2023-04-05 04:58:59,617 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/700a9f652732416db63f544a0eb6408d as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/700a9f652732416db63f544a0eb6408d 2023-04-05 04:58:59,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1680670739608, max timestamp=1680670739618Num uniques 1035; 2023-04-05 04:58:59,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=13050, suffix heap size=135608,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=285 2023-04-05 04:58:59,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1680670739608, max timestamp=1680670739618Num uniques 1035; 2023-04-05 04:58:59,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=13050, suffix heap size=135608,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=285 2023-04-05 04:58:59,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1680670739608, max timestamp=1680670739618Num uniques 1035; 2023-04-05 04:58:59,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=13050, suffix heap size=135608,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=285 2023-04-05 04:58:59,625 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/700a9f652732416db63f544a0eb6408d, entries=150, sequenceid=226, filesize=11.8 K 2023-04-05 04:58:59,625 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=60.38 KB/61830 for ae76f0bbec3b73d99b1c36f6bc76ae4f in 864ms, sequenceid=226, compaction requested=false 2023-04-05 04:58:59,625 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:58:59,625 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:59,628 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:59,628 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:59,628 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,628 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:59,628 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:59,629 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,629 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:59,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:59,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:59,629 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:59,629 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:59,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:59,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:59,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:59,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:59,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:58:59,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:59,631 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ae76f0bbec3b73d99b1c36f6bc76ae4f 3/3 column families, dataSize=67.09 KB heapSize=173.95 KB 2023-04-05 04:58:59,631 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:59,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d35c67c[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:59,631 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:59,631 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:59,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:59,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:59,631 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:59,631 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:59,631 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:59,632 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=A 2023-04-05 04:58:59,632 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:58:59,632 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=B 2023-04-05 04:58:59,632 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,632 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:59,632 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:58:59,632 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=C 2023-04-05 04:58:59,632 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:58:59,639 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:59,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:59,639 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:59,639 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:59,639 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:59,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:59,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:59,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:59,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:59,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:59,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:59,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:59,642 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:59,642 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:59,642 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:59,642 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:59,642 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:59,654 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:59,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:59,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:59,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:59,654 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:59,654 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:59,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:59,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:59,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:59,654 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:59,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:59,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:59,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:59,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:59,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:59,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,656 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:59,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:59,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:59,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38064 deadline: 1680670799654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,659 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,659 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:59,659 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:59,659 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:59,659 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:59,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:59,659 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:59,659 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:59,659 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:59,659 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:59,659 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:59,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:59,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:59,659 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:59,659 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:59,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:59,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:59,661 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:59,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38086 deadline: 1680670799658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,661 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:59,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38100 deadline: 1680670799659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,759 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:59,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:59,760 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:59,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38064 deadline: 1680670799759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:59,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:59,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:59,762 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:59,762 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:59,762 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:59,762 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:59,762 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:59,762 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:59,763 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:59,763 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:59,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:59,763 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,763 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:59,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:59,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:59,764 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:59,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38086 deadline: 1680670799762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:59,764 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:59,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:59,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38100 deadline: 1680670799762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,764 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:59,764 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:59,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:59,765 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:59,765 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:59,765 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:59,765 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:59,765 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:59,765 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:59,765 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:58:59,765 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:58:59,765 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:59,765 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:59,765 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:59,765 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,765 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:59,765 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:59,766 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:59,766 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:59,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:59,766 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:59,766 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:59,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:59,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:59,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:59,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:59,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:59,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b96fe[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:59,767 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:59,767 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:59,767 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:59,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:59,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:59,767 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:59,768 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,768 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:59,768 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:59,768 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:59,768 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,768 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:59,768 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:59,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:59,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:59,768 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:59,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:59,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:59,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:59,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:59,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:59,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:59,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:59,770 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:59,770 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:59,770 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:59,770 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:59,770 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:59,771 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:59,771 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:59,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:59,771 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:59,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:59,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:59,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:59,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:59,771 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:59,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:59,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:59,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:59,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:59,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:59,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:59,772 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:59,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:59,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,773 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,773 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:59,773 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:59,773 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:59,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:59,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:59,773 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:59,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:59,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:59,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:59,774 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,773 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:59,774 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:59,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:59,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:59,774 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:59,774 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:59,774 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:59,774 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:59,874 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:59,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:59,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:59,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:59,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:59,877 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:59,877 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:59,877 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:59,877 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:59,877 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:59,877 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:59,878 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:59,878 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:59,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:59,878 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,878 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:59,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:59,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:59,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:59,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:59,879 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:59,879 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:59,879 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:59,879 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:59,879 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:59,879 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:59,879 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:59,879 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:59,879 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:59,879 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:58:59,879 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:58:59,879 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:59,879 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:59,879 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:59,879 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,880 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:59,880 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:59,880 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,880 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:59,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:59,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:59,880 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:59,880 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:59,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:59,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:59,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:59,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:59,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:59,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ed96196[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:59,881 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:59,882 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:59,882 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:59,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:59,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:59,882 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:59,882 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:59,882 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:59,882 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,882 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:58:59,883 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,883 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:58:59,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:59,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:59,883 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:59,883 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:58:59,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:59,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:59,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:59,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:59,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:59,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:58:59,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:58:59,885 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:58:59,885 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:58:59,885 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:58:59,885 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:58:59,885 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:58:59,886 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:59,886 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:59,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:59,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:59,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:59,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:59,886 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:59,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:59,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:59,886 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:59,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:59,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:59,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:59,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:59,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:59,887 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:59,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:59,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,888 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,888 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:59,888 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:59,888 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:59,888 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:58:59,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:59,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:59,888 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:58:59,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:58:59,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:58:59,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:58:59,888 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,889 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:59,889 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:59,889 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:59,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:58:59,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:58:59,889 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:58:59,889 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:58:59,963 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:59,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38064 deadline: 1680670799962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,966 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:59,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38086 deadline: 1680670799965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,966 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:58:59,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38100 deadline: 1680670799966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,989 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:58:59,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:58:59,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:58:59,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:58:59,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:58:59,993 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:58:59,993 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:59,994 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:58:59,994 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:58:59,994 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:59,994 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:59,995 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,995 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:58:59,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:58:59,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:58:59,995 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:59,995 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:58:59,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:58:59,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:58:59,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:58:59,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:58:59,996 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:58:59,996 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:58:59,996 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:58:59,996 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:58:59,996 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:58:59,996 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:58:59,996 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:58:59,996 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:59,996 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:58:59,996 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:58:59,996 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:58:59,996 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:58:59,996 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:58:59,996 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,996 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:58:59,996 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:58:59,997 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:59,997 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:58:59,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:58:59,997 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:59,997 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:58:59,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:58:59,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:58:59,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:58:59,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:58:59,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:58:59,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@143cdabd[Count = 0] remaining members to acquire global barrier 2023-04-05 04:58:59,999 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:58:59,999 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:59,999 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:59,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:59,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:58:59,999 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:58:59,999 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:58:59,999 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:58:59,999 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:58:59,999 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:00,000 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:00,000 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,000 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:00,001 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:00,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:00,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:00,002 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:00,002 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:00,002 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:00,002 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:00,002 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:00,004 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,004 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:00,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,004 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:00,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:00,004 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:00,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:00,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:00,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:00,005 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:00,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,006 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,007 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,007 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:00,007 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:00,007 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,007 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,007 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:00,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:00,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:00,007 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,007 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,007 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:00,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:00,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:59:00,007 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:00,007 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:00,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:00,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:00,069 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304055dfae42ca5f643929e077d231d494ce9 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304055dfae42ca5f643929e077d231d494ce9 2023-04-05 04:59:00,073 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=495, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/e824346f0be34c4fa519b14295be47d9 2023-04-05 04:59:00,107 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:00,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:00,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:00,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:00,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:00,110 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:00,110 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:00,110 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:00,110 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:00,111 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,111 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,111 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:00,111 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:00,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:00,111 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,111 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:00,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:00,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:00,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:00,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:00,112 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:00,112 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:00,112 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:00,112 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:00,112 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:00,112 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:00,112 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:00,113 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:00,113 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:00,113 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:00,113 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:00,113 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,113 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:00,113 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:00,113 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,113 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:00,114 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,114 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:00,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:00,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7759f860[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:00,115 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:00,115 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,116 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,116 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:00,116 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:00,116 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,116 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:00,116 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:00,117 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:00,117 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:00,117 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:00,117 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:00,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:00,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:00,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:00,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:00,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:00,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:00,119 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:00,119 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:00,119 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:00,119 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:00,119 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:00,120 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,120 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:00,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,120 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:00,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:00,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:00,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:00,121 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:00,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:00,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,122 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:00,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,123 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:00,124 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:00,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:00,124 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:00,123 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:00,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:00,124 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:00,124 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,124 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,124 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,124 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,124 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,124 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:00,124 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:00,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:00,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:00,224 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:00,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:00,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:00,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:00,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:00,227 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:00,227 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:00,227 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:00,227 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:00,227 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,227 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,228 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:00,228 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:00,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:00,228 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,228 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:00,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:00,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,229 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:00,229 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:00,229 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:00,229 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:00,229 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:00,229 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:00,229 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:00,229 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:00,229 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:00,229 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:00,229 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:00,229 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:00,229 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:00,229 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:00,229 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,229 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:00,229 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:00,230 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,230 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:00,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:00,230 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,230 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:00,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:00,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:00,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:00,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:00,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b891c13[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:00,231 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:00,231 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,232 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,232 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:00,232 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:00,232 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:00,232 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,232 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:00,233 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,233 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:00,233 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:00,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:00,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:00,233 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:00,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:00,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:00,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:00,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:00,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:00,235 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:00,235 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:00,235 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:00,235 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:00,235 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:00,236 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,236 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:00,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:00,236 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:00,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:00,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:00,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:00,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,236 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:00,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,237 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:00,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,239 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,239 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,239 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:00,239 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:00,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:00,239 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:00,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:00,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:00,239 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:00,239 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,239 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,239 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,239 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,239 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:00,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:00,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:00,239 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:00,267 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:00,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38064 deadline: 1680670800266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:00,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38100 deadline: 1680670800267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,270 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:00,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38086 deadline: 1680670800269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,339 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:00,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:00,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:00,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:00,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:00,342 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:00,342 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:00,342 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:00,342 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:00,342 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,342 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,343 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,343 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:00,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:00,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:00,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,344 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,344 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:00,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:00,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,344 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:00,344 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:00,344 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:00,344 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:00,344 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:00,344 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:00,344 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:00,345 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:00,345 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:00,345 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:00,345 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:00,345 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:00,345 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:00,345 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:00,345 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,345 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:00,345 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:00,346 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,346 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:00,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:00,346 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,346 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:00,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:00,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:00,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:00,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:00,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ef91d25[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:00,351 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:00,351 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,352 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,352 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:00,352 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:00,352 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:00,352 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,352 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:00,352 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,352 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:00,352 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:00,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:00,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:00,353 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:00,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:00,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:00,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:00,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:00,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:00,357 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:00,357 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:00,357 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:00,357 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:00,357 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:00,358 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,358 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,358 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:00,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:00,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:00,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:00,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:00,359 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:00,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,359 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:00,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:00,361 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:00,361 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,362 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:00,362 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:00,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:00,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:00,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:00,362 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:00,362 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,362 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:00,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:59:00,362 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:00,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:00,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:00,362 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,362 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,362 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,362 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,462 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:00,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:00,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:00,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:00,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:00,466 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:00,466 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:00,466 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:00,466 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:00,466 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,466 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,467 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:00,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:00,467 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:00,467 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,468 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:00,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:00,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,468 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:00,468 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:00,468 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:00,468 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:00,468 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:00,469 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:00,469 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:00,469 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:00,469 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:00,469 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:00,469 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:00,469 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:00,469 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:00,469 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,469 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:00,469 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:00,470 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,470 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:00,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:00,470 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,470 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:00,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:00,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:00,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:00,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:00,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ce2d775[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:00,471 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:00,471 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,472 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,472 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:00,472 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:00,472 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:00,472 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,472 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:00,473 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,473 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:00,473 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:00,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:00,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:00,473 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:00,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:00,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:00,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:00,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:00,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:00,475 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:00,475 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:00,475 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:00,475 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:00,475 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:00,476 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,476 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,476 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:00,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:00,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:00,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:00,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:00,477 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:00,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:00,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,478 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:00,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,480 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,480 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,480 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,480 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,480 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,480 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:00,480 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,480 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,480 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:00,480 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:00,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:00,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:59:00,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:00,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:00,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:00,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:00,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:00,481 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:00,481 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:00,485 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=495 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/cb090e9a856949789a6fb0e640607e7b 2023-04-05 04:59:00,581 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:00,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:00,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:00,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:00,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:00,584 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:00,584 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:00,584 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:00,584 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:00,584 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:00,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38124 deadline: 1680670800582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,584 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,585 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,585 DEBUG [Thread-3533] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4130 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f., hostname=jenkins-hbase20.apache.org,38091,1680670569206, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 04:59:00,585 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,585 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:00,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:00,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:00,586 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,586 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:00,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:00,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,586 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:00,587 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:00,587 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:00,587 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:00,587 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:00,587 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:00,587 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:00,587 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:00,587 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:00,587 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:00,587 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:00,587 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:00,587 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:00,587 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:00,587 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,587 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:00,587 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:00,588 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,588 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:00,588 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,588 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:00,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:00,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44c0ba71[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:00,589 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:00,589 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,590 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,590 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:00,590 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,590 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:00,590 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:00,590 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:00,591 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,591 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:00,591 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:00,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:00,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:00,591 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:00,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:00,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:00,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:00,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:00,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:00,592 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:00,592 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:00,592 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:00,592 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:00,592 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:00,593 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,593 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:00,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,593 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:00,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:00,594 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:00,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,595 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:00,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,596 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,597 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:00,597 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,597 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:00,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:00,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:00,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:00,597 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,597 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:00,597 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:00,597 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:00,597 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:00,597 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,597 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,597 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:00,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:00,697 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:00,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:00,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:00,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:00,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:00,699 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:00,700 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:00,700 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:00,700 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:00,700 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,700 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,701 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:00,701 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:00,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:00,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,701 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,701 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:00,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:00,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,702 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:00,702 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:00,702 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:00,702 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:00,702 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:00,702 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:00,702 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:00,702 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:00,702 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:00,702 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:00,702 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:00,702 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:00,702 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:00,703 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:00,703 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,703 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:00,703 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:00,703 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,703 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:00,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:00,704 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,704 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:00,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:00,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:00,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:00,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:00,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27ca4f7e[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:00,704 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:00,705 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,705 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,705 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:00,705 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:00,705 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:00,705 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,705 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:00,706 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:00,706 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:00,706 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,706 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:00,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:00,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:00,707 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:00,707 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:00,707 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:00,707 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:00,708 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:00,708 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,708 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:00,708 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:00,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:00,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:00,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:00,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:00,709 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:00,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,710 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:00,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,712 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:00,712 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:00,712 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,712 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,712 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,712 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,712 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,712 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:59:00,712 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:00,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:00,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:00,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:00,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:00,712 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:00,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:00,713 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:00,713 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:00,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:00,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38064 deadline: 1680670800772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:00,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38100 deadline: 1680670800773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,776 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:00,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38086 deadline: 1680670800775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,813 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:00,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:00,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:00,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:00,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:00,819 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:00,819 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:00,819 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:00,819 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:00,820 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,820 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,820 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:00,820 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:00,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:00,821 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,821 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:00,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:00,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,821 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:00,821 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:00,822 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:00,822 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:00,822 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:00,822 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:00,822 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:00,822 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:00,822 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:00,822 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:00,822 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:00,822 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:00,822 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:00,822 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:00,822 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,822 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:00,822 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:00,823 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,823 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:00,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:00,823 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,823 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:00,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:00,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:00,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:00,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:00,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@619c7e97[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:00,824 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:00,824 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,824 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,825 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:00,825 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,825 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:00,825 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:00,825 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:00,825 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,825 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:00,825 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:00,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:00,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:00,826 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:00,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:00,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:00,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:00,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:00,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:00,827 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:00,828 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:00,828 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:00,828 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:00,828 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:00,828 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,828 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:00,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:00,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,828 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:00,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:00,829 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:00,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:00,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:00,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,830 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:00,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:00,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,831 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,831 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,831 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,831 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,831 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,831 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:00,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:00,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:00,831 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:00,832 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:59:00,831 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,831 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:00,832 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:00,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:00,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:00,832 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:00,832 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:00,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:00,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:00,894 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=495 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/8b0b2565269149419153b2254e39c1e6 2023-04-05 04:59:00,897 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/e824346f0be34c4fa519b14295be47d9 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/e824346f0be34c4fa519b14295be47d9 2023-04-05 04:59:00,900 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/e824346f0be34c4fa519b14295be47d9, entries=150, sequenceid=495, filesize=25.4 K 2023-04-05 04:59:00,900 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/cb090e9a856949789a6fb0e640607e7b as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/cb090e9a856949789a6fb0e640607e7b 2023-04-05 04:59:00,904 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/cb090e9a856949789a6fb0e640607e7b, entries=150, sequenceid=495, filesize=11.9 K 2023-04-05 04:59:00,904 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/8b0b2565269149419153b2254e39c1e6 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/8b0b2565269149419153b2254e39c1e6 2023-04-05 04:59:00,907 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/8b0b2565269149419153b2254e39c1e6, entries=150, sequenceid=495, filesize=11.9 K 2023-04-05 04:59:00,908 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~213.75 KB/218880, currentSize=100.63 KB/103050 for ae76f0bbec3b73d99b1c36f6bc76ae4f in 1276ms, sequenceid=495, compaction requested=true 2023-04-05 04:59:00,908 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:59:00,908 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:59:00,908 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:59:00,908 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:59:00,908 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:59:00,908 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:59:00,909 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78078 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:59:00,909 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36498 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:59:00,909 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): ae76f0bbec3b73d99b1c36f6bc76ae4f/A is initiating minor compaction (all files) 2023-04-05 04:59:00,909 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): ae76f0bbec3b73d99b1c36f6bc76ae4f/B is initiating minor compaction (all files) 2023-04-05 04:59:00,909 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ae76f0bbec3b73d99b1c36f6bc76ae4f/A in TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:00,909 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ae76f0bbec3b73d99b1c36f6bc76ae4f/B in TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:00,909 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/3ca36fdff31f4b948c8436c5207a560c, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/7a92851f34504a2495e92ba4ff7039fe, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/e824346f0be34c4fa519b14295be47d9] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp, totalSize=76.2 K 2023-04-05 04:59:00,909 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/8b6d1e6220cd433a9b51858476ab2c47, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/335531c356a040a4ae376e7039d56e91, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/cb090e9a856949789a6fb0e640607e7b] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp, totalSize=35.6 K 2023-04-05 04:59:00,909 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 3ca36fdff31f4b948c8436c5207a560c, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1680670736450 2023-04-05 04:59:00,909 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 8b6d1e6220cd433a9b51858476ab2c47, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1680670736450 2023-04-05 04:59:00,910 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 7a92851f34504a2495e92ba4ff7039fe, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=226, earliestPutTs=1680670737038 2023-04-05 04:59:00,910 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 335531c356a040a4ae376e7039d56e91, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=226, earliestPutTs=1680670737038 2023-04-05 04:59:00,910 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting e824346f0be34c4fa519b14295be47d9, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=495, earliestPutTs=1680670739623 2023-04-05 04:59:00,911 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting cb090e9a856949789a6fb0e640607e7b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=495, earliestPutTs=1680670739623 2023-04-05 04:59:00,917 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): ae76f0bbec3b73d99b1c36f6bc76ae4f#B#compaction#488 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:59:00,919 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): ae76f0bbec3b73d99b1c36f6bc76ae4f#A#compaction#489 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:59:00,924 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/768c871f9c9744efa06646a3a7f40d6e as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/768c871f9c9744efa06646a3a7f40d6e 2023-04-05 04:59:00,927 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ae76f0bbec3b73d99b1c36f6bc76ae4f/B of ae76f0bbec3b73d99b1c36f6bc76ae4f into 768c871f9c9744efa06646a3a7f40d6e(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:59:00,927 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:59:00,927 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f., storeName=ae76f0bbec3b73d99b1c36f6bc76ae4f/B, priority=13, startTime=1680670740908; duration=0sec 2023-04-05 04:59:00,928 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:59:00,928 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-05 04:59:00,928 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36498 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-05 04:59:00,928 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): ae76f0bbec3b73d99b1c36f6bc76ae4f/C is initiating minor compaction (all files) 2023-04-05 04:59:00,928 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ae76f0bbec3b73d99b1c36f6bc76ae4f/C in TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:00,929 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/e513e2369c194d1babad81a3c9900670, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/700a9f652732416db63f544a0eb6408d, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/8b0b2565269149419153b2254e39c1e6] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp, totalSize=35.6 K 2023-04-05 04:59:00,929 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting e513e2369c194d1babad81a3c9900670, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1680670736450 2023-04-05 04:59:00,929 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 700a9f652732416db63f544a0eb6408d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=226, earliestPutTs=1680670737038 2023-04-05 04:59:00,929 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 8b0b2565269149419153b2254e39c1e6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=495, earliestPutTs=1680670739623 2023-04-05 04:59:00,932 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:00,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:00,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:00,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:00,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:00,935 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): ae76f0bbec3b73d99b1c36f6bc76ae4f#C#compaction#490 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:59:00,935 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:00,936 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:00,936 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:00,936 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:00,936 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,936 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,936 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:00,936 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:00,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:00,937 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:00,937 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:00,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:00,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:00,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:00,937 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:00,942 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:00,942 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:00,942 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:00,942 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:00,942 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:00,944 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:00,944 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:00,944 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:00,944 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:00,944 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ae76f0bbec3b73d99b1c36f6bc76ae4f 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-05 04:59:00,944 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=A 2023-04-05 04:59:00,945 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:59:00,945 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=B 2023-04-05 04:59:00,945 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:59:00,945 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=C 2023-04-05 04:59:00,945 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:59:00,949 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/a1f3106e10ad4d25b3cdadc93e2905e7 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/a1f3106e10ad4d25b3cdadc93e2905e7 2023-04-05 04:59:00,953 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ae76f0bbec3b73d99b1c36f6bc76ae4f/C of ae76f0bbec3b73d99b1c36f6bc76ae4f into a1f3106e10ad4d25b3cdadc93e2905e7(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:59:00,954 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:59:00,954 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f., storeName=ae76f0bbec3b73d99b1c36f6bc76ae4f/C, priority=13, startTime=1680670740908; duration=0sec 2023-04-05 04:59:00,954 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:59:00,960 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040580224e8b5c454b23b0d77b7527feafc8 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040580224e8b5c454b23b0d77b7527feafc8 2023-04-05 04:59:00,988 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=514, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/68835a4d706b4c1695628647066a00ee 2023-04-05 04:59:01,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:59:01,085 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:01,108 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:01,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38120 deadline: 1680670801107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:01,210 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:01,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38120 deadline: 1680670801209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:01,396 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=514 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/4349c68abb9b460384fe3414103c621e 2023-04-05 04:59:01,405 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=514 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/19e1d8f3155d4e96a5b8fd7545329a17 2023-04-05 04:59:01,409 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/68835a4d706b4c1695628647066a00ee as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/68835a4d706b4c1695628647066a00ee 2023-04-05 04:59:01,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:01,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38120 deadline: 1680670801411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:01,412 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/68835a4d706b4c1695628647066a00ee, entries=150, sequenceid=514, filesize=25.4 K 2023-04-05 04:59:01,413 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/4349c68abb9b460384fe3414103c621e as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/4349c68abb9b460384fe3414103c621e 2023-04-05 04:59:01,416 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/4349c68abb9b460384fe3414103c621e, entries=150, sequenceid=514, filesize=11.9 K 2023-04-05 04:59:01,417 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/19e1d8f3155d4e96a5b8fd7545329a17 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/19e1d8f3155d4e96a5b8fd7545329a17 2023-04-05 04:59:01,419 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/19e1d8f3155d4e96a5b8fd7545329a17, entries=150, sequenceid=514, filesize=11.9 K 2023-04-05 04:59:01,420 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for ae76f0bbec3b73d99b1c36f6bc76ae4f in 476ms, sequenceid=514, compaction requested=false 2023-04-05 04:59:01,420 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:59:01,420 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:01,420 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:01,420 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:01,420 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:01,420 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:01,420 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:01,421 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:01,421 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:01,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:01,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:01,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:01,421 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:01,421 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:01,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:01,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:01,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:01,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:01,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:01,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:01,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@baf44ae[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:01,422 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:01,422 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:01,423 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:01,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:01,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:01,423 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:01,423 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:01,423 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:01,424 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:01,424 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:01,424 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:01,424 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:01,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:01,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:01,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:01,424 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:01,424 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:01,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:01,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:01,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:01,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:01,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:01,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:01,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:01,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:01,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:01,426 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:01,426 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:01,426 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:01,426 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:01,426 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:01,427 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:01,427 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:01,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:01,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:01,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:01,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:01,427 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:01,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:01,428 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:01,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:01,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:01,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:01,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:01,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:01,429 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:01,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:01,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:01,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:01,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:01,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:01,430 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:01,430 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:01,430 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:01,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:01,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:01,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:01,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:01,430 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:01,430 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:01,430 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:01,430 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:01,430 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:01,431 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:01,430 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:01,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:01,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:01,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:59:01,431 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:01,431 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:01,531 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:01,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:01,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:01,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:01,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:01,533 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:01,533 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:01,533 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:01,533 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:01,533 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:01,533 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:01,534 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:01,534 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:01,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:01,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:01,534 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:01,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:01,534 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:01,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:01,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:01,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:01,535 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:01,535 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:01,535 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:01,535 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:01,535 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:01,535 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:01,535 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:01,535 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:01,535 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:01,535 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:01,535 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ae76f0bbec3b73d99b1c36f6bc76ae4f 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-05 04:59:01,536 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=A 2023-04-05 04:59:01,536 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:59:01,536 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=B 2023-04-05 04:59:01,536 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:59:01,536 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=C 2023-04-05 04:59:01,536 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:59:01,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:59:01,715 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:01,742 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/96e47bfa486b47a6b04e84eb1637e08a as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/96e47bfa486b47a6b04e84eb1637e08a 2023-04-05 04:59:01,746 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ae76f0bbec3b73d99b1c36f6bc76ae4f/A of ae76f0bbec3b73d99b1c36f6bc76ae4f into 96e47bfa486b47a6b04e84eb1637e08a(size=25.8 K), total size for store is 51.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:59:01,746 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:59:01,746 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f., storeName=ae76f0bbec3b73d99b1c36f6bc76ae4f/A, priority=13, startTime=1680670740908; duration=0sec 2023-04-05 04:59:01,746 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:59:01,752 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:01,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38120 deadline: 1680670801751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:01,780 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:01,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38086 deadline: 1680670801779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:01,780 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:01,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38064 deadline: 1680670801780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:01,784 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:01,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38100 deadline: 1680670801783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:01,855 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:01,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38120 deadline: 1680670801853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:01,945 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304051ce339ad7f144ad296970ec8b5efb9b8 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304051ce339ad7f144ad296970ec8b5efb9b8 2023-04-05 04:59:01,949 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=529, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/5791a60dc4ad4e9dacf4e113a4927b98 2023-04-05 04:59:02,058 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:02,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38120 deadline: 1680670802057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,356 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=529 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/4920cf1c7a04433fbffdb6c98d5a66b2 2023-04-05 04:59:02,360 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:02,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38120 deadline: 1680670802359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,363 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=529 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/bd94f1d302e6433fb430c2736c86e9ff 2023-04-05 04:59:02,366 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/5791a60dc4ad4e9dacf4e113a4927b98 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/5791a60dc4ad4e9dacf4e113a4927b98 2023-04-05 04:59:02,369 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/5791a60dc4ad4e9dacf4e113a4927b98, entries=150, sequenceid=529, filesize=25.4 K 2023-04-05 04:59:02,369 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/4920cf1c7a04433fbffdb6c98d5a66b2 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/4920cf1c7a04433fbffdb6c98d5a66b2 2023-04-05 04:59:02,375 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/4920cf1c7a04433fbffdb6c98d5a66b2, entries=150, sequenceid=529, filesize=11.9 K 2023-04-05 04:59:02,376 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/bd94f1d302e6433fb430c2736c86e9ff as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/bd94f1d302e6433fb430c2736c86e9ff 2023-04-05 04:59:02,379 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/bd94f1d302e6433fb430c2736c86e9ff, entries=150, sequenceid=529, filesize=11.9 K 2023-04-05 04:59:02,379 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for ae76f0bbec3b73d99b1c36f6bc76ae4f in 844ms, sequenceid=529, compaction requested=true 2023-04-05 04:59:02,380 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:59:02,380 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:02,380 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:02,382 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:02,382 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,382 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:02,382 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:02,383 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:02,383 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:02,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:02,383 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:02,384 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:02,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:02,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:02,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:02,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:02,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:02,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a7d7498[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:02,385 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:02,385 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:02,385 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:02,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:02,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:02,385 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:02,385 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:02,386 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:02,386 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,386 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:02,386 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,386 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:02,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:02,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:02,386 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:02,386 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:02,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:02,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:02,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:02,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:02,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:02,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:02,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:02,388 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:02,388 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:02,388 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:02,388 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:02,388 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:02,389 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:02,389 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:02,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:02,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:02,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:02,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:02,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:02,389 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:02,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:02,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:02,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:02,390 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:02,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:02,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:02,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:02,391 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:02,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:02,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,392 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:02,392 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:02,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:02,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:02,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:02,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:02,393 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,393 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:02,393 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:02,393 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:02,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:59:02,393 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:02,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:02,393 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:02,393 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:02,394 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:02,394 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:02,394 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:02,494 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:02,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:02,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:02,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:02,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:02,496 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:02,496 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:02,496 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:02,496 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:02,497 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:02,497 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:02,497 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:02,497 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:02,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:02,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:02,498 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,498 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:02,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:02,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:02,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:02,498 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:02,498 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:02,498 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:02,499 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:02,499 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:02,499 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:02,499 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:02,499 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:02,499 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:02,499 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:02,499 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ae76f0bbec3b73d99b1c36f6bc76ae4f 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-05 04:59:02,499 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=A 2023-04-05 04:59:02,499 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:59:02,499 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=B 2023-04-05 04:59:02,499 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:59:02,499 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=C 2023-04-05 04:59:02,499 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:59:02,540 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405ce09544f32154b7db49a721a5dcea8db to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405ce09544f32154b7db49a721a5dcea8db 2023-04-05 04:59:02,548 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=548, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/9157d49340e64eb1b7d6d7bece16491a 2023-04-05 04:59:02,556 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=548 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/a37ae1a2edf04942a6c36b164c25a762 2023-04-05 04:59:02,569 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=548 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/3f244370122e40e6aaf384de4ad6e8b2 2023-04-05 04:59:02,572 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/9157d49340e64eb1b7d6d7bece16491a as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/9157d49340e64eb1b7d6d7bece16491a 2023-04-05 04:59:02,575 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/9157d49340e64eb1b7d6d7bece16491a, entries=150, sequenceid=548, filesize=25.4 K 2023-04-05 04:59:02,578 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/a37ae1a2edf04942a6c36b164c25a762 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/a37ae1a2edf04942a6c36b164c25a762 2023-04-05 04:59:02,581 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/a37ae1a2edf04942a6c36b164c25a762, entries=150, sequenceid=548, filesize=11.9 K 2023-04-05 04:59:02,584 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/3f244370122e40e6aaf384de4ad6e8b2 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/3f244370122e40e6aaf384de4ad6e8b2 2023-04-05 04:59:02,597 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/3f244370122e40e6aaf384de4ad6e8b2, entries=150, sequenceid=548, filesize=11.9 K 2023-04-05 04:59:02,598 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for ae76f0bbec3b73d99b1c36f6bc76ae4f in 99ms, sequenceid=548, compaction requested=true 2023-04-05 04:59:02,598 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:59:02,598 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:02,598 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:02,598 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:02,598 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,598 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:02,598 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:02,600 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,600 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:02,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:02,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:02,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:02,600 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:02,600 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:02,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:02,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:02,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:02,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:02,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f397316[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:02,601 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:02,601 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:02,602 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:02,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:02,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:02,602 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:02,602 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:02,602 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:02,602 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,602 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:02,603 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:02,603 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,603 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:02,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:02,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:02,603 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:02,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:02,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:02,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:02,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:02,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:02,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:02,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:02,605 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:02,605 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:02,605 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:02,605 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:02,605 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:02,606 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:02,606 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:02,606 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:02,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:02,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:02,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:02,606 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:02,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:02,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:02,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:02,607 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:02,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:02,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,609 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:02,609 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,609 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:02,609 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:02,609 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,609 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:02,609 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:02,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:02,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:02,609 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:02,609 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:02,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:59:02,609 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:02,609 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:02,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:02,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:02,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:59:02,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:02,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:02,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:02,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:02,709 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:02,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:02,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:02,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:02,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:02,712 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:02,712 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:02,712 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:02,712 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:02,712 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:02,712 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:02,713 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:02,713 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:02,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:02,713 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,713 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:02,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:02,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:02,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:02,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:02,714 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:02,714 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:02,714 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:02,714 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:02,714 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:02,714 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:02,714 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:02,714 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:02,714 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:02,714 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:02,715 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:59:02,715 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:02,715 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:02,715 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:02,715 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,715 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:02,715 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:02,716 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,716 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:02,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:02,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:02,716 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:02,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:02,716 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:02,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:02,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:02,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:02,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:02,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d53caa0[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:02,717 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:02,717 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:02,718 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:02,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:02,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:02,718 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:02,718 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:02,718 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,718 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:02,718 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:02,719 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,719 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:02,719 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:02,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:02,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:02,719 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:02,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:02,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:02,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:02,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:02,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:02,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:02,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:02,721 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:02,721 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:02,721 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:02,721 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:02,721 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:02,722 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:02,722 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:02,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:02,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:02,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:02,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:02,722 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:02,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:02,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:02,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:02,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:02,723 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:02,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:02,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:02,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:02,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,724 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:02,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:02,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,725 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,725 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:02,726 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:02,726 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,726 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:02,726 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:02,726 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:02,726 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:02,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:02,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:02,726 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:02,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:02,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:02,726 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:02,726 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:02,726 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:02,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:02,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:02,826 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:02,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:02,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:02,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:02,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:02,829 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:02,829 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:02,829 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:02,830 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:02,830 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:02,830 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:02,831 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:02,831 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:02,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:02,831 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:02,831 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:02,832 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:02,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:02,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:02,832 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:02,832 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:02,832 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:02,832 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:02,832 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:02,832 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:02,833 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:02,833 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:02,833 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:02,833 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:02,834 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:59:02,834 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:02,834 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:02,834 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:02,834 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,834 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:02,834 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:02,835 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,835 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:02,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:02,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:02,835 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:02,836 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:02,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:02,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:02,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:02,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:02,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:02,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c286390[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:02,837 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:02,837 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:02,837 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:02,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:02,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:02,838 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:02,838 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:02,838 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:02,838 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,838 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:02,839 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:02,839 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:02,839 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:02,839 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:02,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:02,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:02,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:02,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:02,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:02,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:02,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:02,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:02,841 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:02,841 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:02,841 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:02,841 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:02,841 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:02,842 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:02,842 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:02,842 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:02,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:02,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:02,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:02,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:02,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:02,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:02,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:02,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:02,843 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:02,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:02,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:02,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:02,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:02,845 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:02,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,846 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:02,846 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:02,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:02,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:02,846 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:02,846 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:02,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:02,847 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:02,847 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:02,847 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,847 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:02,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:02,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:02,847 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:02,847 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:02,847 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:02,847 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:02,947 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:02,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:02,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:02,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:02,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:02,951 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:02,951 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:02,951 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:02,951 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:02,951 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:02,951 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:02,952 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:02,952 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:02,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:02,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:02,952 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:02,952 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:02,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:02,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:02,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:02,953 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:02,966 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:02,966 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:02,966 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:02,966 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:02,966 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:02,966 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:02,966 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:02,967 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:02,967 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ae76f0bbec3b73d99b1c36f6bc76ae4f 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-05 04:59:02,968 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=A 2023-04-05 04:59:02,968 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:59:02,968 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=B 2023-04-05 04:59:02,968 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:59:02,968 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=C 2023-04-05 04:59:02,968 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:59:02,970 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:02,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:59:03,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670743009, max timestamp=1680670743030Num uniques 859; 2023-04-05 04:59:03,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670743009, max timestamp=1680670743030Num uniques 859; 2023-04-05 04:59:03,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680670743009, max timestamp=1680670743030Num uniques 859; 2023-04-05 04:59:03,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:59:03,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:59:03,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-05 04:59:03,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680670743102, max timestamp=1680670743138Num uniques 1018; 2023-04-05 04:59:03,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-05 04:59:03,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680670743102, max timestamp=1680670743138Num uniques 1018; 2023-04-05 04:59:03,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-05 04:59:03,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680670743102, max timestamp=1680670743138Num uniques 1018; 2023-04-05 04:59:03,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-05 04:59:03,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680670743206, max timestamp=1680670743220Num uniques 1027; 2023-04-05 04:59:03,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-05 04:59:03,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680670743206, max timestamp=1680670743220Num uniques 1027; 2023-04-05 04:59:03,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-05 04:59:03,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680670743206, max timestamp=1680670743220Num uniques 1027; 2023-04-05 04:59:03,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-05 04:59:03,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680670743262, max timestamp=1680670743279Num uniques 1036; 2023-04-05 04:59:03,282 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-05 04:59:03,282 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680670743262, max timestamp=1680670743279Num uniques 1036; 2023-04-05 04:59:03,282 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-05 04:59:03,282 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680670743262, max timestamp=1680670743279Num uniques 1036; 2023-04-05 04:59:03,282 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-05 04:59:03,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680670743320, max timestamp=1680670743340Num uniques 1045; 2023-04-05 04:59:03,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680670743320, max timestamp=1680670743340Num uniques 1045; 2023-04-05 04:59:03,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680670743320, max timestamp=1680670743340Num uniques 1045; 2023-04-05 04:59:03,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-05 04:59:03,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-05 04:59:03,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-05 04:59:03,399 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304051867afe2980f4eac997831fa6a4db23e to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304051867afe2980f4eac997831fa6a4db23e 2023-04-05 04:59:03,403 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=559, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/de651601efbc4d62b58a98f9b899516d 2023-04-05 04:59:03,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:03,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38120 deadline: 1680670803416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:03,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:03,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38120 deadline: 1680670803520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:03,724 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:03,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38120 deadline: 1680670803723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:03,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:03,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38086 deadline: 1680670803783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:03,785 DEBUG [Thread-3527] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4126 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f., hostname=jenkins-hbase20.apache.org,38091,1680670569206, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 04:59:03,790 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:03,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38064 deadline: 1680670803788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:03,791 DEBUG [Thread-3529] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4136 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f., hostname=jenkins-hbase20.apache.org,38091,1680670569206, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 04:59:03,806 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:03,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38100 deadline: 1680670803804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:03,807 DEBUG [Thread-3531] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4149 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f., hostname=jenkins-hbase20.apache.org,38091,1680670569206, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 04:59:03,819 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=559 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/472ac4f9f09f4708809ccb832e86328e 2023-04-05 04:59:03,826 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=559 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/73036436b39846348760b41a086346d9 2023-04-05 04:59:03,829 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/de651601efbc4d62b58a98f9b899516d as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/de651601efbc4d62b58a98f9b899516d 2023-04-05 04:59:03,831 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/de651601efbc4d62b58a98f9b899516d, entries=150, sequenceid=559, filesize=25.4 K 2023-04-05 04:59:03,832 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/472ac4f9f09f4708809ccb832e86328e as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/472ac4f9f09f4708809ccb832e86328e 2023-04-05 04:59:03,834 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/472ac4f9f09f4708809ccb832e86328e, entries=150, sequenceid=559, filesize=11.9 K 2023-04-05 04:59:03,835 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/73036436b39846348760b41a086346d9 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/73036436b39846348760b41a086346d9 2023-04-05 04:59:03,837 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/73036436b39846348760b41a086346d9, entries=150, sequenceid=559, filesize=11.9 K 2023-04-05 04:59:03,838 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for ae76f0bbec3b73d99b1c36f6bc76ae4f in 871ms, sequenceid=559, compaction requested=true 2023-04-05 04:59:03,838 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:59:03,838 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:03,838 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:03,838 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:03,838 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:03,838 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:03,838 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:03,839 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:03,839 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:03,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:03,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:03,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:03,839 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:03,839 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:03,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:03,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:03,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:03,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:03,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:03,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:03,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d500739[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:03,840 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:03,840 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:03,842 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:03,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:03,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:03,842 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:03,842 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:03,842 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:03,842 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:03,842 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:03,842 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:03,842 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:03,842 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:03,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:03,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:03,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:03,842 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:03,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:03,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:03,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:03,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:03,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:03,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:03,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:03,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:03,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:03,844 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:03,844 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:03,844 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:03,844 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:03,844 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:03,845 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:03,845 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:03,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:03,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:03,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:03,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:03,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:03,845 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:03,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:03,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:03,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:03,846 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:03,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:03,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:03,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:03,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:03,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:03,847 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:03,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:03,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:03,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:03,848 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:03,848 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:03,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:03,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:03,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:03,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:03,848 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:03,848 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:03,849 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:03,849 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:03,849 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:03,849 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:03,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:03,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:03,849 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:03,849 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:03,849 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:03,849 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:03,949 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:03,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:03,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:03,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:03,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:03,951 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:03,951 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:03,951 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:03,951 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:03,952 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:03,952 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:03,952 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:03,952 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:03,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:03,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:03,953 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:03,953 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:03,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:03,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:03,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:03,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:03,953 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:03,953 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:03,953 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:03,953 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:03,953 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:03,954 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:03,954 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:03,954 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:03,954 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:03,954 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:03,954 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ae76f0bbec3b73d99b1c36f6bc76ae4f 3/3 column families, dataSize=140.89 KB heapSize=401.53 KB 2023-04-05 04:59:03,954 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=A 2023-04-05 04:59:03,954 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:59:03,954 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=B 2023-04-05 04:59:03,954 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:59:03,954 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=C 2023-04-05 04:59:03,954 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:59:04,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:59:04,028 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:04,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:04,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38120 deadline: 1680670804051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:04,156 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:04,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38120 deadline: 1680670804154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:04,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:04,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38120 deadline: 1680670804357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:04,364 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304053b2978d8050e4c6481f66750c2812ed4 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304053b2978d8050e4c6481f66750c2812ed4 2023-04-05 04:59:04,598 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:04,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38124 deadline: 1680670804596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:04,599 DEBUG [Thread-3533] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8144 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f., hostname=jenkins-hbase20.apache.org,38091,1680670569206, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 04:59:04,663 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:04,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38120 deadline: 1680670804661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:04,768 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=665, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/ff49ad34719c4184812632266d9e3136 2023-04-05 04:59:05,168 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:05,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38120 deadline: 1680670805166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:05,179 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=665 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/4af017fb1b134c458f672d4e68c562af 2023-04-05 04:59:05,187 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=665 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/bcf0e035cc214af0afaa5723e4c66141 2023-04-05 04:59:05,189 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/ff49ad34719c4184812632266d9e3136 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/ff49ad34719c4184812632266d9e3136 2023-04-05 04:59:05,192 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/ff49ad34719c4184812632266d9e3136, entries=150, sequenceid=665, filesize=25.4 K 2023-04-05 04:59:05,192 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/4af017fb1b134c458f672d4e68c562af as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/4af017fb1b134c458f672d4e68c562af 2023-04-05 04:59:05,195 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/4af017fb1b134c458f672d4e68c562af, entries=150, sequenceid=665, filesize=11.9 K 2023-04-05 04:59:05,196 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/bcf0e035cc214af0afaa5723e4c66141 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/bcf0e035cc214af0afaa5723e4c66141 2023-04-05 04:59:05,198 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/bcf0e035cc214af0afaa5723e4c66141, entries=150, sequenceid=665, filesize=11.9 K 2023-04-05 04:59:05,199 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~401.48 KB/411120, currentSize=40.25 KB/41220 for ae76f0bbec3b73d99b1c36f6bc76ae4f in 1245ms, sequenceid=665, compaction requested=true 2023-04-05 04:59:05,199 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:59:05,199 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:05,199 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:05,199 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:05,199 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:05,199 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:05,199 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:05,200 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:05,200 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:05,200 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:05,202 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:05,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:05,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:05,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:05,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:05,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:05,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:05,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:05,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:05,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:05,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19857846[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:05,203 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:05,203 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:05,204 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:05,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:05,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:05,204 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:05,204 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:05,204 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:05,205 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:05,205 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:05,205 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:05,205 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:05,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:05,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:05,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:05,205 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:05,205 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:05,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:05,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:05,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:05,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:05,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:05,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:05,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:05,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:05,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:05,207 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:05,207 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:05,207 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:05,207 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:05,207 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:05,208 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:05,208 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:05,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:05,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:05,208 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:05,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:05,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:05,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:05,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:05,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:05,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:05,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:05,209 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:05,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:05,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:05,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:05,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:05,211 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:05,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:05,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:05,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:05,212 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:05,212 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:05,212 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:05,212 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:05,212 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:05,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:05,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:05,212 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:05,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:05,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:05,212 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:05,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:05,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:05,213 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:05,213 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:05,212 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:05,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:59:05,212 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:05,213 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:05,313 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:05,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:05,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:05,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:05,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:05,315 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:05,315 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:05,315 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:05,315 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:05,316 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:05,316 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:05,316 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:05,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:05,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:05,316 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:05,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:05,317 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:05,317 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:05,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:05,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:05,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:05,317 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:05,317 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:05,317 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:05,317 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:05,317 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:05,317 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:05,317 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:05,317 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:05,318 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:05,318 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:05,318 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ae76f0bbec3b73d99b1c36f6bc76ae4f 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-05 04:59:05,318 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=A 2023-04-05 04:59:05,318 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:59:05,318 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=B 2023-04-05 04:59:05,318 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:59:05,318 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=C 2023-04-05 04:59:05,318 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:59:05,334 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405d58d0a0d78b44972899bbb4478dbc8d8 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405d58d0a0d78b44972899bbb4478dbc8d8 2023-04-05 04:59:05,340 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=674, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/cf6858a296a449b5a27cca01ce97fbb4 2023-04-05 04:59:05,747 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=674 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/a631481bb6d748d580037f2ce466cdd7 2023-04-05 04:59:06,155 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=674 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/3d045cb70a814a67bc06cdf38c4113a6 2023-04-05 04:59:06,158 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/cf6858a296a449b5a27cca01ce97fbb4 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/cf6858a296a449b5a27cca01ce97fbb4 2023-04-05 04:59:06,160 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/cf6858a296a449b5a27cca01ce97fbb4, entries=150, sequenceid=674, filesize=25.4 K 2023-04-05 04:59:06,161 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/a631481bb6d748d580037f2ce466cdd7 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/a631481bb6d748d580037f2ce466cdd7 2023-04-05 04:59:06,164 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/a631481bb6d748d580037f2ce466cdd7, entries=150, sequenceid=674, filesize=11.9 K 2023-04-05 04:59:06,164 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/3d045cb70a814a67bc06cdf38c4113a6 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/3d045cb70a814a67bc06cdf38c4113a6 2023-04-05 04:59:06,167 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/3d045cb70a814a67bc06cdf38c4113a6, entries=150, sequenceid=674, filesize=11.9 K 2023-04-05 04:59:06,167 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=0 B/0 for ae76f0bbec3b73d99b1c36f6bc76ae4f in 849ms, sequenceid=674, compaction requested=true 2023-04-05 04:59:06,167 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:59:06,168 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:06,168 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:06,168 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:06,168 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,168 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:06,168 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:06,169 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,169 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:06,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:06,169 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,169 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:06,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:06,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:06,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:06,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:06,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3aa0a71c[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:06,178 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:06,178 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,184 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,184 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:06,190 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,190 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:06,190 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:06,190 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:06,191 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:06,191 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:06,191 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:06,191 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:06,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:06,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:06,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:06,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:06,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:06,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:06,194 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:06,195 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:06,195 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:06,195 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:06,195 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:06,196 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,196 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:06,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,196 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:06,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:06,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:06,197 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:06,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:06,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:06,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,198 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:06,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,199 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,199 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:06,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:06,199 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:06,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:06,199 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:06,199 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:06,200 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:59:06,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:06,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:06,200 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:06,200 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:06,200 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:06,200 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,200 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,200 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:06,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:06,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:59:06,261 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ae76f0bbec3b73d99b1c36f6bc76ae4f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-05 04:59:06,261 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=A 2023-04-05 04:59:06,261 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:59:06,261 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=B 2023-04-05 04:59:06,261 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:59:06,261 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=C 2023-04-05 04:59:06,261 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:59:06,300 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:06,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:06,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:06,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:06,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:06,304 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:06,304 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:06,304 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:06,304 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:06,304 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,304 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:06,305 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:06,305 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:06,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:06,305 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,305 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:06,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:06,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:06,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:06,306 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:06,307 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:06,307 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:06,307 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:06,307 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:06,307 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:06,307 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:06,307 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:06,307 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:06,307 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:06,307 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:06,307 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:06,307 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:06,307 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,307 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:06,307 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:06,308 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,308 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:06,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:06,309 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,309 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:06,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:06,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:06,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:06,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:06,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@359b673b[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:06,310 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:06,310 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,310 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,311 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:06,311 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:06,311 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,311 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:06,311 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:06,312 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:06,312 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,313 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:06,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:06,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:06,313 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:06,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:06,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:06,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:06,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:06,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:06,314 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:06,315 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:06,315 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:06,315 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:06,315 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:06,316 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,316 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:06,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:06,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,316 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:06,317 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:06,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:06,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:06,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:06,317 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:06,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,319 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,319 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:06,319 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:06,319 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:06,319 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:06,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:06,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:06,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:06,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:06,319 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,319 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,319 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,319 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:06,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:59:06,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:06,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:06,319 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,320 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:06,320 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:06,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670746296, max timestamp=1680670746328Num uniques 859; 2023-04-05 04:59:06,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:59:06,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670746296, max timestamp=1680670746328Num uniques 859; 2023-04-05 04:59:06,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:59:06,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680670746296, max timestamp=1680670746328Num uniques 859; 2023-04-05 04:59:06,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-05 04:59:06,392 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:06,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38120 deadline: 1680670806389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,420 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:06,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:06,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:06,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:06,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:06,423 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:06,423 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:06,423 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:06,423 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:06,423 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,423 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:06,424 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,424 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:06,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:06,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:06,424 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,424 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:06,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:06,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:06,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:06,424 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:06,426 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:06,426 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:06,426 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:06,426 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:06,426 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:06,426 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:06,426 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:06,426 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:06,426 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:06,426 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:06,426 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:06,426 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:06,426 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:06,427 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,427 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:06,427 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:06,427 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,428 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:06,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:06,428 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,428 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:06,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:06,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:06,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:06,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:06,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@589fd77e[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:06,429 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:06,429 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,429 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,429 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:06,430 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:06,430 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,430 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:06,430 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:06,430 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,430 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:06,430 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:06,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:06,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:06,430 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:06,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:06,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:06,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:06,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:06,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:06,432 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:06,432 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:06,432 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:06,432 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:06,432 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:06,433 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,433 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,433 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:06,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:06,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:06,433 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:06,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:06,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:06,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:06,434 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:06,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,436 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,436 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:06,436 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:06,436 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:06,436 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:06,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:06,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:06,436 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,436 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,436 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:06,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:06,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:06,436 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:06,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:06,437 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:06,437 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:06,437 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:59:06,496 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:06,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38120 deadline: 1680670806493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,537 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:06,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:06,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:06,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:06,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:06,540 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:06,540 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:06,540 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:06,540 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:06,540 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,540 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:06,541 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:06,541 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:06,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:06,541 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,541 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:06,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:06,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:06,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:06,542 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:06,542 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:06,542 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:06,542 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:06,542 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:06,542 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:06,542 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:06,542 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:06,542 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:06,543 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:06,543 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:06,543 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:06,543 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:06,543 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:06,543 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,543 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:06,543 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:06,544 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,544 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:06,544 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,544 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:06,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:06,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:06,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:06,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:06,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:06,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a6ac363[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:06,545 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:06,545 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,545 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,546 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:06,546 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:06,546 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:06,546 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,546 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:06,546 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,546 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:06,547 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:06,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:06,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:06,547 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:06,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:06,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:06,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:06,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:06,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:06,549 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:06,549 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:06,549 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:06,549 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:06,549 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:06,550 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,550 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:06,550 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:06,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:06,550 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:06,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:06,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:06,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:06,551 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:06,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,552 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,552 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:06,552 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:06,552 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:06,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:06,552 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:06,553 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:06,553 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,553 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:06,553 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:06,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:06,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:06,553 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:06,553 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,553 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:06,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:06,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:59:06,653 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:06,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:06,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:06,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:06,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:06,656 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:06,656 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:06,656 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:06,656 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:06,656 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,656 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:06,657 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:06,657 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:06,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:06,657 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,657 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:06,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:06,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:06,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:06,657 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:06,658 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:06,658 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:06,658 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:06,658 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:06,658 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:06,658 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:06,658 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:06,658 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:06,658 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:06,658 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:06,658 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:06,658 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:06,658 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:06,658 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,658 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:06,658 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:06,659 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,659 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:06,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:06,659 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,659 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:06,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:06,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:06,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:06,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:06,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@554d1175[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:06,660 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:06,661 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,661 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,661 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:06,661 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,661 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:06,661 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:06,661 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:06,662 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,662 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:06,662 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:06,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:06,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:06,662 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:06,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:06,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:06,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:06,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:06,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:06,664 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:06,664 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:06,664 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:06,664 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:06,664 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:06,665 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,665 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,665 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:06,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:06,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:06,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:06,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:06,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:06,665 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:06,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,666 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:06,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,667 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,667 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:06,667 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:06,667 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:06,667 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:06,667 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:06,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:06,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:06,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:06,667 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:06,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:59:06,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:06,668 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:06,667 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,668 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:06,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:06,668 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,668 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,674 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304055d42d5f6847d4dd18ec64d4ad876c049 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304055d42d5f6847d4dd18ec64d4ad876c049 2023-04-05 04:59:06,678 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=684, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/e91b81194ae14dac94eb8a31954597dc 2023-04-05 04:59:06,700 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:06,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38120 deadline: 1680670806698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,768 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:06,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:06,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:06,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:06,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:06,771 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:06,771 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:06,771 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:06,771 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:06,771 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,771 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:06,772 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:06,772 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:06,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:06,772 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,772 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:06,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:06,772 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:06,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:06,772 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:06,773 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:06,773 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:06,773 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:06,773 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:06,773 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:06,773 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:06,773 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:06,773 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:06,773 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:06,773 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:06,773 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:06,773 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:06,773 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:06,773 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,773 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:06,773 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:06,774 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,774 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:06,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:06,774 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,774 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:06,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:06,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:06,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:06,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:06,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b811658[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:06,775 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:06,775 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,776 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,776 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:06,776 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:06,776 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:06,776 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,776 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:06,776 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,776 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:06,777 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:06,777 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:06,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:06,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:06,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:06,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:06,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:06,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:06,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:06,779 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:06,779 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:06,779 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:06,779 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:06,779 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:06,780 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,780 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,780 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:06,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:06,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:06,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,780 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:06,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:06,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:06,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:06,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,781 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:06,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,782 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,783 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:06,783 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:06,783 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:06,783 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:06,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:06,783 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,783 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,783 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,783 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:06,783 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:06,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:06,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:06,783 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:06,783 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:06,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:06,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:06,784 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:59:06,883 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:06,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:06,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:06,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:06,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:06,886 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:06,886 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:06,886 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:06,886 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:06,886 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,886 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:06,887 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:06,887 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:06,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:06,887 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,887 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:06,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:06,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:06,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:06,888 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:06,888 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:06,888 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:06,888 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:06,888 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:06,888 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:06,888 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:06,888 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:06,888 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:06,888 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:06,888 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:06,888 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:06,888 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:06,888 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:06,888 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,888 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:06,888 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:06,889 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,889 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:06,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:06,889 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,889 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:06,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:06,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:06,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:06,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:06,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d830271[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:06,891 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:06,891 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,891 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,891 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,891 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:06,891 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:06,891 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:06,891 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:06,892 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,892 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:06,892 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:06,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:06,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:06,892 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:06,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:06,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:06,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:06,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:06,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:06,894 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:06,894 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:06,894 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:06,894 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:06,894 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:06,894 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,894 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:06,895 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:06,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:06,895 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:06,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:06,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:06,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:06,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,896 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:06,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:06,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,897 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,897 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:06,897 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:06,897 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:06,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:59:06,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:06,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:06,898 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:06,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:06,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:06,897 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:06,898 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:06,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:06,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:06,898 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,898 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:06,898 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:06,898 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:06,898 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:06,998 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:06,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:07,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:07,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:07,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:07,001 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:07,001 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:07,001 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:07,001 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:07,001 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,001 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,002 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,002 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:07,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:07,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:07,002 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,002 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:07,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:07,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,003 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:07,003 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:07,003 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:07,003 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:07,003 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:07,003 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:07,003 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:07,003 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:07,003 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:07,003 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:07,003 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:07,003 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:07,003 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:07,003 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:07,003 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,003 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:07,003 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:07,004 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:07,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:07,004 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,004 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:07,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38120 deadline: 1680670807001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:07,004 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,005 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:07,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:07,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:07,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:07,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21cebb79[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:07,005 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:07,006 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,006 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,006 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:07,006 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:07,006 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,006 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:07,006 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:07,007 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:07,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:07,007 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:07,007 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:07,007 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:07,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:07,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:07,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:07,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:07,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:07,009 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:07,009 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:07,009 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:07,009 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:07,009 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:07,009 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,009 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,009 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:07,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:07,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:07,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:07,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:07,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:07,010 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:07,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,011 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:07,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,013 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,013 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:07,013 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,013 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:07,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:07,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:07,013 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:07,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:07,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:07,013 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:07,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:07,014 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,014 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:07,014 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,014 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:07,014 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,014 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,085 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=684 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/10bbafa6ddbf492583fdf0dab26b2e5f 2023-04-05 04:59:07,114 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:07,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:07,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:07,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:07,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:07,117 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:07,117 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:07,117 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:07,117 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:07,117 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,117 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,118 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,118 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:07,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:07,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:07,118 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,118 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:07,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:07,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,119 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:07,119 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:07,119 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:07,119 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:07,119 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:07,119 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:07,119 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:07,119 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:07,119 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:07,119 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:07,119 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:07,119 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:07,119 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:07,119 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:07,119 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,119 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:07,119 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:07,120 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,120 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:07,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:07,120 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,120 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:07,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:07,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:07,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:07,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:07,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cb5d08c[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:07,121 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:07,121 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,122 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,122 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,122 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:07,122 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:07,122 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:07,122 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:07,123 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:07,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:07,123 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:07,123 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:07,123 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:07,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:07,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:07,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:07,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:07,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:07,125 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:07,125 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:07,125 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:07,125 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:07,125 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:07,125 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,126 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:07,126 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:07,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:07,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:07,126 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:07,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:07,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:07,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,127 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:07,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,129 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,129 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,129 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:07,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:07,129 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:07,129 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:07,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:59:07,129 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:07,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:07,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:07,129 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,129 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,129 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,129 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,129 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:07,130 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:07,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:07,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:07,230 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:07,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:07,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:07,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:07,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:07,232 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:07,232 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:07,232 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:07,232 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:07,233 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,233 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,233 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:07,233 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:07,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:07,233 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,234 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:07,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:07,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,234 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:07,234 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:07,234 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:07,234 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:07,234 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:07,234 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:07,234 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:07,234 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:07,234 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:07,234 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:07,235 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:07,235 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:07,235 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:07,235 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:07,235 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,235 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:07,235 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:07,235 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,235 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:07,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:07,235 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,235 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:07,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:07,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:07,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:07,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:07,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f353694[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:07,236 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:07,237 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,237 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,237 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:07,237 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:07,237 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:07,237 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,237 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:07,238 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:07,238 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,238 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:07,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:07,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:07,238 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:07,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:07,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:07,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:07,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:07,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:07,240 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:07,240 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:07,240 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:07,240 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:07,240 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:07,240 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,240 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:07,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,240 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:07,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:07,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,241 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:07,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:07,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:07,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:07,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,242 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:07,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,243 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,243 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:07,243 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:07,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:07,243 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:07,243 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:07,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:07,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:07,243 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,243 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,243 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,243 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,243 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,243 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:07,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:59:07,243 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:07,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:07,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:07,344 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:07,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:07,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:07,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:07,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:07,346 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:07,346 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:07,346 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:07,346 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:07,347 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,347 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,347 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:07,347 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:07,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:07,347 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,348 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:07,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:07,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,348 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:07,348 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:07,348 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:07,348 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:07,348 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:07,348 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:07,348 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:07,348 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:07,348 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:07,349 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:07,349 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:07,349 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:07,349 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:07,349 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:07,349 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,349 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:07,349 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:07,349 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,349 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:07,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:07,350 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,350 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:07,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:07,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:07,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:07,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:07,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a203989[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:07,351 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:07,351 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,351 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,352 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:07,352 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,352 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:07,352 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:07,352 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:07,352 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,352 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:07,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:07,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:07,352 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:07,353 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:07,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:07,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:07,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:07,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:07,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:07,355 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:07,355 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:07,355 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:07,356 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:07,356 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:07,356 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,356 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,356 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:07,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:07,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:07,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:07,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:07,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:07,357 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:07,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,358 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:07,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,360 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,360 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,360 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,360 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,360 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,360 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:07,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:59:07,360 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:07,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:07,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:07,360 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,360 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:07,360 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:07,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:07,360 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:07,360 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:07,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:07,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:07,460 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:07,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:07,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:07,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:07,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:07,463 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:07,463 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:07,463 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:07,463 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:07,463 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,463 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,464 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,464 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:07,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:07,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:07,464 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,465 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:07,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:07,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,465 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:07,465 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:07,465 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:07,465 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:07,465 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:07,465 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:07,465 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:07,465 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:07,466 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:07,466 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:07,466 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:07,466 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:07,466 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:07,466 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:07,466 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,466 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:07,466 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:07,467 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,467 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:07,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:07,467 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,467 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:07,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:07,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:07,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:07,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:07,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ee15ac7[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:07,468 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:07,468 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,468 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,469 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:07,469 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,469 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:07,469 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:07,469 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:07,469 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,469 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:07,469 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:07,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:07,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:07,470 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:07,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:07,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:07,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:07,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:07,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:07,471 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:07,471 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:07,471 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:07,471 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:07,471 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:07,472 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,472 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:07,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,472 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:07,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:07,472 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:07,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:07,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:07,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:07,473 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:07,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,474 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,475 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,475 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,475 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:07,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:07,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:07,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:07,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:07,475 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:07,475 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:07,475 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:07,475 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,475 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:07,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:59:07,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:07,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:07,475 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,475 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,475 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,493 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=684 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/2a171f4627634618951c647eb9e2039e 2023-04-05 04:59:07,496 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/e91b81194ae14dac94eb8a31954597dc as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/e91b81194ae14dac94eb8a31954597dc 2023-04-05 04:59:07,498 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/e91b81194ae14dac94eb8a31954597dc, entries=250, sequenceid=684, filesize=39.2 K 2023-04-05 04:59:07,499 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/10bbafa6ddbf492583fdf0dab26b2e5f as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/10bbafa6ddbf492583fdf0dab26b2e5f 2023-04-05 04:59:07,501 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/10bbafa6ddbf492583fdf0dab26b2e5f, entries=100, sequenceid=684, filesize=9.5 K 2023-04-05 04:59:07,502 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/2a171f4627634618951c647eb9e2039e as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/2a171f4627634618951c647eb9e2039e 2023-04-05 04:59:07,504 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/2a171f4627634618951c647eb9e2039e, entries=100, sequenceid=684, filesize=9.5 K 2023-04-05 04:59:07,505 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for ae76f0bbec3b73d99b1c36f6bc76ae4f in 1244ms, sequenceid=684, compaction requested=true 2023-04-05 04:59:07,505 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:59:07,505 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:59:07,505 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:59:07,505 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-05 04:59:07,505 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:59:07,505 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-05 04:59:07,508 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 222790 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-05 04:59:07,508 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 95386 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-05 04:59:07,508 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): ae76f0bbec3b73d99b1c36f6bc76ae4f/A is initiating minor compaction (all files) 2023-04-05 04:59:07,508 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): ae76f0bbec3b73d99b1c36f6bc76ae4f/B is initiating minor compaction (all files) 2023-04-05 04:59:07,509 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ae76f0bbec3b73d99b1c36f6bc76ae4f/A in TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:07,509 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ae76f0bbec3b73d99b1c36f6bc76ae4f/B in TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:07,509 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/96e47bfa486b47a6b04e84eb1637e08a, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/68835a4d706b4c1695628647066a00ee, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/5791a60dc4ad4e9dacf4e113a4927b98, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/9157d49340e64eb1b7d6d7bece16491a, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/de651601efbc4d62b58a98f9b899516d, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/ff49ad34719c4184812632266d9e3136, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/cf6858a296a449b5a27cca01ce97fbb4, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/e91b81194ae14dac94eb8a31954597dc] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp, totalSize=217.6 K 2023-04-05 04:59:07,509 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/768c871f9c9744efa06646a3a7f40d6e, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/4349c68abb9b460384fe3414103c621e, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/4920cf1c7a04433fbffdb6c98d5a66b2, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/a37ae1a2edf04942a6c36b164c25a762, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/472ac4f9f09f4708809ccb832e86328e, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/4af017fb1b134c458f672d4e68c562af, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/a631481bb6d748d580037f2ce466cdd7, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/10bbafa6ddbf492583fdf0dab26b2e5f] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp, totalSize=93.2 K 2023-04-05 04:59:07,509 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 96e47bfa486b47a6b04e84eb1637e08a, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=495, earliestPutTs=1680670739623 2023-04-05 04:59:07,509 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 768c871f9c9744efa06646a3a7f40d6e, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=495, earliestPutTs=1680670739623 2023-04-05 04:59:07,509 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 68835a4d706b4c1695628647066a00ee, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=514, earliestPutTs=1680670739652 2023-04-05 04:59:07,509 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 4349c68abb9b460384fe3414103c621e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=514, earliestPutTs=1680670739652 2023-04-05 04:59:07,510 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 4920cf1c7a04433fbffdb6c98d5a66b2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=529, earliestPutTs=1680670741085 2023-04-05 04:59:07,510 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 5791a60dc4ad4e9dacf4e113a4927b98, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=529, earliestPutTs=1680670741085 2023-04-05 04:59:07,510 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting a37ae1a2edf04942a6c36b164c25a762, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1680670741744 2023-04-05 04:59:07,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:59:07,510 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 9157d49340e64eb1b7d6d7bece16491a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1680670741744 2023-04-05 04:59:07,510 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 472ac4f9f09f4708809ccb832e86328e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=559, earliestPutTs=1680670742929 2023-04-05 04:59:07,510 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ae76f0bbec3b73d99b1c36f6bc76ae4f 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-05 04:59:07,510 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=A 2023-04-05 04:59:07,511 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting de651601efbc4d62b58a98f9b899516d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=559, earliestPutTs=1680670742929 2023-04-05 04:59:07,511 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:59:07,511 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 4af017fb1b134c458f672d4e68c562af, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=665, earliestPutTs=1680670743403 2023-04-05 04:59:07,511 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=B 2023-04-05 04:59:07,511 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:59:07,511 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=C 2023-04-05 04:59:07,511 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting ff49ad34719c4184812632266d9e3136, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=665, earliestPutTs=1680670743403 2023-04-05 04:59:07,511 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:59:07,511 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting a631481bb6d748d580037f2ce466cdd7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=674, earliestPutTs=1680670744028 2023-04-05 04:59:07,511 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting cf6858a296a449b5a27cca01ce97fbb4, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=674, earliestPutTs=1680670744028 2023-04-05 04:59:07,511 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 10bbafa6ddbf492583fdf0dab26b2e5f, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=684, earliestPutTs=1680670746253 2023-04-05 04:59:07,511 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting e91b81194ae14dac94eb8a31954597dc, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=684, earliestPutTs=1680670746236 2023-04-05 04:59:07,525 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405661e0c5a542743c1b62c11f0db2c0804 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405661e0c5a542743c1b62c11f0db2c0804 2023-04-05 04:59:07,532 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): ae76f0bbec3b73d99b1c36f6bc76ae4f#B#compaction#514 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:59:07,534 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): ae76f0bbec3b73d99b1c36f6bc76ae4f#A#compaction#513 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:59:07,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:07,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38120 deadline: 1680670807540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,575 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:07,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:07,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:07,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:07,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:07,578 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:07,578 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:07,578 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:07,578 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:07,578 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,578 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,579 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:07,579 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:07,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:07,579 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,580 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:07,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:07,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,580 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:07,580 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:07,580 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:07,580 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:07,580 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:07,580 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:07,580 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:07,580 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:07,580 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:07,580 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:07,580 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:07,580 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:07,580 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:07,581 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:07,581 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,581 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:07,581 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:07,581 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,581 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:07,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:07,582 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,582 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:07,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:07,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:07,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:07,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:07,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c62c427[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:07,583 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:07,583 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,583 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,584 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:07,584 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:07,584 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:07,584 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,584 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:07,584 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:07,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:07,584 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:07,584 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:07,584 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:07,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:07,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:07,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:07,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:07,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:07,586 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:07,586 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:07,586 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:07,586 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:07,586 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:07,587 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,587 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:07,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:07,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,587 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:07,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:07,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:07,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:07,587 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:07,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,588 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:07,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,589 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:07,589 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:07,590 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:07,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:07,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:07,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:07,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:07,590 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,590 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,590 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,590 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:07,590 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,590 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:07,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:07,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:07,590 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,590 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,590 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,647 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:07,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38120 deadline: 1680670807645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,690 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:07,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:07,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:07,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:07,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:07,693 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:07,693 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:07,693 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:07,693 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:07,693 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,693 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,694 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:07,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:07,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:07,694 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,694 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,694 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:07,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:07,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,695 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:07,695 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:07,695 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:07,695 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:07,695 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:07,695 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:07,695 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:07,695 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:07,695 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:07,695 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:07,695 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:07,695 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:07,695 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:07,695 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:07,695 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,695 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:07,695 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:07,696 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,696 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:07,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:07,696 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,696 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:07,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:07,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:07,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:07,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:07,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ae5daa9[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:07,697 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:07,697 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,698 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,698 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:07,698 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:07,698 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:07,698 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,698 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:07,699 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,699 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:07,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:07,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:07,699 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:07,699 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:07,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:07,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:07,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:07,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:07,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:07,700 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:07,701 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:07,701 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:07,701 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:07,701 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:07,701 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,701 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:07,701 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:07,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:07,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:07,702 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:07,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:07,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:07,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,703 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:07,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,704 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:07,704 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,704 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:07,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:59:07,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:07,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:07,704 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,704 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:07,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:07,704 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:07,704 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,704 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:07,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:07,704 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,704 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,705 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,705 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:07,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:07,798 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:07,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38086 deadline: 1680670807796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,799 DEBUG [Thread-3527] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8140 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f., hostname=jenkins-hbase20.apache.org,38091,1680670569206, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 04:59:07,804 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:07,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:07,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:07,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:07,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:07,808 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:07,808 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:07,808 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:07,808 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:07,808 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,808 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,809 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:07,809 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:07,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:07,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,809 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,809 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:07,809 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:07,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,810 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:07,810 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:07,810 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:07,810 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:07,810 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:07,810 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:07,810 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:07,810 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:07,810 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:07,810 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:07,811 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:07,811 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:07,811 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:07,811 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:07,811 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,811 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:07,811 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:07,811 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:07,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38100 deadline: 1680670807809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,813 DEBUG [Thread-3531] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8154 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f., hostname=jenkins-hbase20.apache.org,38091,1680670569206, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 04:59:07,813 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,813 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:07,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:07,813 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:07,813 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:07,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:07,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:07,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:07,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61224e0d[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:07,814 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:07,815 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,815 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,815 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,815 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:07,815 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:07,815 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:07,815 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:07,816 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,816 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:07,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:07,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:07,816 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:07,816 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:07,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:07,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:07,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:07,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:07,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:07,817 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:07,817 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:07,817 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:07,817 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:07,817 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:07,818 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,818 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,818 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:07,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:07,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:07,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:07,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:07,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:07,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38064 deadline: 1680670807813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,819 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:07,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,820 DEBUG [Thread-3529] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8165 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f., hostname=jenkins-hbase20.apache.org,38091,1680670569206, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-05 04:59:07,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:07,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,820 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:07,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,822 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,822 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,822 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,822 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,822 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:07,822 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,822 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:07,822 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:07,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:07,822 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:07,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:07,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:07,822 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:07,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:59:07,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:07,822 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:07,822 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:07,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:07,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38120 deadline: 1680670807848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,923 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:07,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:07,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:07,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:07,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:07,926 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:07,926 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:07,926 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:07,926 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:07,926 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,927 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,927 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:07,927 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:07,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:07,927 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,927 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:07,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:07,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,928 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:07,928 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:07,928 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:07,928 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:07,928 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:07,928 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:07,928 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:07,928 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:07,928 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:07,929 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:07,929 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:07,929 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:07,929 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:07,929 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,929 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:07,929 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:07,929 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,929 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:07,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:07,930 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,930 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:07,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:07,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:07,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:07,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:07,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15e1178c[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:07,931 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:07,931 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,931 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,932 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:07,932 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:07,932 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:07,932 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,932 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:07,932 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,932 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:07,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:07,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:07,932 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:07,932 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:07,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:07,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:07,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:07,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:07,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:07,934 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:07,934 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:07,934 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:07,934 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:07,934 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:07,934 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=721, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/52681e1a32dd4dbea38ed0949d18edc0 2023-04-05 04:59:07,936 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,936 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:07,936 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:07,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:07,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:07,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:07,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:07,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,936 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:07,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,937 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:07,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:07,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,939 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:07,939 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:07,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:07,939 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:07,939 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,939 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:07,939 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:07,939 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:07,939 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,940 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:07,940 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:07,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:07,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:07,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:07,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:07,940 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:07,940 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:07,942 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/307b20ef9ca14ac592ac9694d666b5b5 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/307b20ef9ca14ac592ac9694d666b5b5 2023-04-05 04:59:07,945 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in ae76f0bbec3b73d99b1c36f6bc76ae4f/B of ae76f0bbec3b73d99b1c36f6bc76ae4f into 307b20ef9ca14ac592ac9694d666b5b5(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:59:07,946 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:59:07,946 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f., storeName=ae76f0bbec3b73d99b1c36f6bc76ae4f/B, priority=8, startTime=1680670747505; duration=0sec 2023-04-05 04:59:07,946 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:59:07,946 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-05 04:59:07,948 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 95386 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-05 04:59:07,948 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): ae76f0bbec3b73d99b1c36f6bc76ae4f/C is initiating minor compaction (all files) 2023-04-05 04:59:07,948 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ae76f0bbec3b73d99b1c36f6bc76ae4f/C in TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:07,948 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/a1f3106e10ad4d25b3cdadc93e2905e7, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/19e1d8f3155d4e96a5b8fd7545329a17, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/bd94f1d302e6433fb430c2736c86e9ff, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/3f244370122e40e6aaf384de4ad6e8b2, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/73036436b39846348760b41a086346d9, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/bcf0e035cc214af0afaa5723e4c66141, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/3d045cb70a814a67bc06cdf38c4113a6, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/2a171f4627634618951c647eb9e2039e] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp, totalSize=93.2 K 2023-04-05 04:59:07,948 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting a1f3106e10ad4d25b3cdadc93e2905e7, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=495, earliestPutTs=1680670739623 2023-04-05 04:59:07,949 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 19e1d8f3155d4e96a5b8fd7545329a17, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=514, earliestPutTs=1680670739652 2023-04-05 04:59:07,949 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting bd94f1d302e6433fb430c2736c86e9ff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=529, earliestPutTs=1680670741085 2023-04-05 04:59:07,949 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 3f244370122e40e6aaf384de4ad6e8b2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1680670741744 2023-04-05 04:59:07,949 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 73036436b39846348760b41a086346d9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=559, earliestPutTs=1680670742929 2023-04-05 04:59:07,950 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting bcf0e035cc214af0afaa5723e4c66141, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=665, earliestPutTs=1680670743403 2023-04-05 04:59:07,950 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 3d045cb70a814a67bc06cdf38c4113a6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=674, earliestPutTs=1680670744028 2023-04-05 04:59:07,950 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 2a171f4627634618951c647eb9e2039e, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=684, earliestPutTs=1680670746253 2023-04-05 04:59:07,960 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): ae76f0bbec3b73d99b1c36f6bc76ae4f#C#compaction#516 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:59:08,040 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:08,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:08,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:08,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:08,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:08,043 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:08,043 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:08,043 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:08,043 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:08,043 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,043 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:08,044 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:08,044 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:08,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:08,044 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,044 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:08,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:08,045 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:08,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:08,045 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:08,045 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:08,045 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:08,045 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:08,045 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:08,045 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:08,045 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:08,045 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:08,045 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:08,046 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:08,046 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:08,046 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:08,046 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:08,046 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:08,046 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,046 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:08,046 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:08,047 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,047 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:08,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:08,047 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,047 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:08,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:08,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:08,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:08,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:08,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ff8baeb[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:08,048 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:08,048 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,049 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,049 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,049 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:08,049 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:08,049 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:08,049 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:08,049 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:08,049 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:08,049 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:08,050 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:08,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:08,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:08,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:08,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:08,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:08,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:08,051 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:08,051 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:08,051 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:08,051 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:08,051 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:08,052 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,052 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,052 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:08,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:08,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:08,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,052 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:08,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:08,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:08,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,053 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:08,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:08,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,054 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,055 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:08,055 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:08,055 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:08,055 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:08,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:08,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:08,055 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,055 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,055 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:08,055 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,055 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:08,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:08,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:08,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:08,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:59:08,055 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:08,055 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:08,152 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:08,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38120 deadline: 1680670808151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,155 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:08,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:08,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:08,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:08,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:08,158 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:08,158 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:08,158 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:08,158 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:08,158 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,159 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:08,159 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,160 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:08,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:08,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:08,160 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,160 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:08,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:08,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:08,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:08,161 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:08,161 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:08,161 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:08,161 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:08,161 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:08,161 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:08,161 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:08,161 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:08,161 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:08,161 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:08,161 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:08,161 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:08,161 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:08,161 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:08,161 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,161 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:08,162 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:08,162 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,162 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:08,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:08,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:08,163 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,163 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:08,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:08,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:08,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:08,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bac012f[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:08,165 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:08,165 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,166 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,166 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:08,166 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:08,166 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:08,167 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,167 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:08,168 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:08,168 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,168 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:08,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:08,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:08,168 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:08,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:08,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:08,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:08,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:08,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:08,170 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:08,170 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:08,170 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:08,170 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:08,170 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:08,170 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,170 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:08,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:08,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,171 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:08,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:08,171 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:08,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:08,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:08,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,172 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:08,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,173 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:08,173 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:08,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:08,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:08,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:59:08,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:08,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:08,173 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,173 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:08,174 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:08,174 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:08,174 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:08,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:08,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:08,174 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:08,174 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,174 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,174 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,174 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,274 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:08,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:08,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:08,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:08,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:08,276 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:08,276 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:08,277 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:08,277 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:08,277 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,277 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:08,277 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,277 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:08,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:08,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:08,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:08,278 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,278 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:08,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:08,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:08,281 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:08,282 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:08,282 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:08,282 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:08,282 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:08,282 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:08,284 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:08,284 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:08,284 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:08,284 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:08,284 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:08,284 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:08,284 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:08,284 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:08,284 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,284 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:08,284 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:08,291 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,291 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:08,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:08,291 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,291 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:08,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:08,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:08,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:08,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:08,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@588d6071[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:08,292 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:08,292 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,295 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,295 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:08,295 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:08,295 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:08,295 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,295 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:08,296 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,296 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:08,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:08,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:08,296 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:08,296 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:08,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:08,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:08,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:08,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:08,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:08,298 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:08,298 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:08,298 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:08,298 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:08,298 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:08,298 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,298 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,298 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:08,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:08,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:08,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:08,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:08,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:08,299 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:08,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,300 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:08,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,301 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,301 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:08,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:08,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:59:08,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:08,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:08,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:08,301 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:08,301 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:08,301 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:08,301 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:08,301 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:08,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:08,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:08,301 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:08,301 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,302 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,302 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,302 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,351 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=721 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/25a4cce421d34a3894e2ed9da46e3e69 2023-04-05 04:59:08,354 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/2d3bcaabd6f74b72a2ba7d57dce1f5a3 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/2d3bcaabd6f74b72a2ba7d57dce1f5a3 2023-04-05 04:59:08,356 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in ae76f0bbec3b73d99b1c36f6bc76ae4f/A of ae76f0bbec3b73d99b1c36f6bc76ae4f into 2d3bcaabd6f74b72a2ba7d57dce1f5a3(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:59:08,357 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:59:08,357 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f., storeName=ae76f0bbec3b73d99b1c36f6bc76ae4f/A, priority=8, startTime=1680670747505; duration=0sec 2023-04-05 04:59:08,357 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:59:08,367 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/e1108cf49bb24ca1b42441ce97313b26 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/e1108cf49bb24ca1b42441ce97313b26 2023-04-05 04:59:08,369 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in ae76f0bbec3b73d99b1c36f6bc76ae4f/C of ae76f0bbec3b73d99b1c36f6bc76ae4f into e1108cf49bb24ca1b42441ce97313b26(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:59:08,370 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:59:08,370 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f., storeName=ae76f0bbec3b73d99b1c36f6bc76ae4f/C, priority=8, startTime=1680670747505; duration=0sec 2023-04-05 04:59:08,370 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:59:08,402 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:08,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:08,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:08,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:08,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:08,404 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:08,404 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:08,404 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:08,404 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:08,404 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,404 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:08,405 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:08,405 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:08,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:08,405 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,405 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:08,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:08,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:08,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:08,406 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:08,406 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:08,406 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:08,406 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:08,406 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:08,406 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:08,406 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:08,406 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:08,406 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:08,406 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:08,407 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:08,407 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:08,407 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:08,407 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:08,407 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,407 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:08,407 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:08,407 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,407 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:08,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:08,408 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,408 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:08,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:08,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:08,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:08,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:08,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@454d519b[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:08,409 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:08,409 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,409 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,409 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:08,409 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:08,409 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:08,410 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,410 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:08,410 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:08,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:08,410 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:08,410 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:08,410 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:08,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:08,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:08,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:08,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:08,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:08,412 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:08,412 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:08,412 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:08,412 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:08,412 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:08,413 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,413 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:08,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:08,413 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:08,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:08,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:08,413 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:08,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,414 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:08,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:08,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,416 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:08,416 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:08,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:08,416 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:08,416 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:08,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:08,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:08,416 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:08,416 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:08,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:08,416 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:08,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:59:08,416 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:08,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:08,416 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,416 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,416 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,416 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,516 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:08,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:08,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:08,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:08,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:08,519 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:08,519 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:08,519 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:08,519 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:08,519 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,520 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:08,520 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:08,520 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:08,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:08,520 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,520 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:08,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:08,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:08,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:08,521 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:08,521 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:08,521 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:08,521 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:08,521 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:08,521 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:08,521 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:08,521 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:08,521 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:08,522 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:08,522 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:08,522 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:08,522 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:08,522 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:08,522 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,522 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:08,522 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:08,522 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,522 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:08,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:08,523 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,523 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:08,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:08,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:08,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:08,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:08,524 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:08,524 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f68943b[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:08,524 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,524 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,524 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:08,524 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:08,524 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:08,524 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:08,525 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,525 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:08,525 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:08,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:08,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:08,525 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:08,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:08,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:08,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:08,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:08,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:08,527 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:08,527 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:08,527 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:08,527 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:08,527 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:08,528 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,528 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:08,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,528 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:08,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:08,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:08,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:08,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:08,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,529 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:08,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,529 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:08,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,531 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,531 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:08,531 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:08,531 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:08,531 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:08,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:08,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:08,531 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:08,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:08,531 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:08,531 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:08,531 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,531 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:59:08,531 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:08,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:08,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:08,531 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,631 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:08,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:08,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:08,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:08,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:08,634 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:08,634 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:08,634 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:08,634 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:08,634 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,634 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:08,635 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:08,635 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:08,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:08,635 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,635 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:08,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:08,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:08,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:08,636 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:08,636 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:08,636 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:08,636 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:08,636 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:08,636 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:08,636 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:08,636 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:08,636 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:08,636 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:08,636 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:08,636 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:08,636 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:08,636 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:08,636 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,636 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:08,636 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:08,637 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,637 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:08,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:08,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:08,638 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,638 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:08,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:08,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:08,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:08,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4064180[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:08,639 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:08,639 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,639 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,639 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:08,639 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:08,639 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:08,639 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,639 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:08,640 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,640 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:08,640 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:08,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:08,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:08,640 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:08,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:08,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:08,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:08,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:08,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:08,642 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:08,642 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:08,642 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:08,642 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:08,642 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:08,643 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,643 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,643 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:08,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:08,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:08,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:08,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:08,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:08,644 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:08,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,644 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:08,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,646 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,646 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:08,646 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:08,646 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:08,646 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:08,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:59:08,646 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:08,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:08,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:08,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:08,646 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:08,646 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,646 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,646 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,646 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:08,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:08,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:08,647 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:08,659 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:08,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38120 deadline: 1680670808657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,747 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:08,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:08,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:08,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:08,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:08,749 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:08,749 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:08,749 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:08,750 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:08,750 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,750 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:08,750 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:08,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:08,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:08,750 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,751 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,751 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:08,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:08,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:08,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:08,751 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:08,751 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:08,751 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:08,751 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:08,751 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:08,752 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:08,752 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:08,752 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:08,752 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:08,752 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:08,752 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:08,752 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:08,752 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:08,752 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:08,752 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,752 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:08,752 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:08,753 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,753 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:08,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:08,753 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,753 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:08,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:08,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:08,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:08,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:08,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@373b3799[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:08,754 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:08,754 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,754 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,754 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:08,754 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:08,754 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:08,754 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,754 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:08,755 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,755 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:08,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:08,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:08,755 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:08,755 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:08,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:08,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:08,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:08,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:08,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:08,757 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:08,757 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:08,757 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:08,757 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:08,757 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:08,757 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,757 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,757 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:08,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:08,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:08,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,758 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:08,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:08,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:08,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:08,759 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=721 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/267117be1c164a4db29cf25dd4ca3b3d 2023-04-05 04:59:08,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,759 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:08,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:08,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,761 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:08,761 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:08,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:08,761 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:08,761 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:08,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:08,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:08,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:08,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:08,761 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/52681e1a32dd4dbea38ed0949d18edc0 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/52681e1a32dd4dbea38ed0949d18edc0 2023-04-05 04:59:08,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:59:08,761 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:08,762 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:08,762 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,762 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,762 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:08,762 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,762 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:08,762 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:08,764 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/52681e1a32dd4dbea38ed0949d18edc0, entries=200, sequenceid=721, filesize=32.3 K 2023-04-05 04:59:08,765 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/25a4cce421d34a3894e2ed9da46e3e69 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/25a4cce421d34a3894e2ed9da46e3e69 2023-04-05 04:59:08,767 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/25a4cce421d34a3894e2ed9da46e3e69, entries=150, sequenceid=721, filesize=11.9 K 2023-04-05 04:59:08,768 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/267117be1c164a4db29cf25dd4ca3b3d as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/267117be1c164a4db29cf25dd4ca3b3d 2023-04-05 04:59:08,770 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/267117be1c164a4db29cf25dd4ca3b3d, entries=150, sequenceid=721, filesize=11.9 K 2023-04-05 04:59:08,771 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for ae76f0bbec3b73d99b1c36f6bc76ae4f in 1261ms, sequenceid=721, compaction requested=false 2023-04-05 04:59:08,771 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:59:08,863 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:08,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:08,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:08,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:08,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:08,865 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:08,865 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:08,865 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:08,865 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:08,865 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,866 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:08,866 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:08,866 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:08,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:08,866 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:08,866 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:08,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:08,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:08,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:08,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:08,867 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:08,867 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:08,867 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:08,867 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:08,867 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:08,867 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:08,867 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:08,867 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:08,867 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:08,867 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:08,868 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ae76f0bbec3b73d99b1c36f6bc76ae4f 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-05 04:59:08,868 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=A 2023-04-05 04:59:08,868 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:59:08,868 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=B 2023-04-05 04:59:08,868 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:59:08,868 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=C 2023-04-05 04:59:08,868 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:59:09,278 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405da9c17c5ba5243a2abf2c71e6a91b951 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405da9c17c5ba5243a2abf2c71e6a91b951 2023-04-05 04:59:09,281 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=733, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/addfbf1037784c1cad2ef311c39e775f 2023-04-05 04:59:09,288 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=733 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/fcc2ceee291b472291f6a3f0d5857888 2023-04-05 04:59:09,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:59:09,666 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:09,695 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=733 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/851c0178c787439d81cc1546c74dbd4a 2023-04-05 04:59:09,698 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/addfbf1037784c1cad2ef311c39e775f as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/addfbf1037784c1cad2ef311c39e775f 2023-04-05 04:59:09,700 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/addfbf1037784c1cad2ef311c39e775f, entries=150, sequenceid=733, filesize=25.4 K 2023-04-05 04:59:09,701 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/fcc2ceee291b472291f6a3f0d5857888 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/fcc2ceee291b472291f6a3f0d5857888 2023-04-05 04:59:09,703 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/fcc2ceee291b472291f6a3f0d5857888, entries=150, sequenceid=733, filesize=11.9 K 2023-04-05 04:59:09,704 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/851c0178c787439d81cc1546c74dbd4a as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/851c0178c787439d81cc1546c74dbd4a 2023-04-05 04:59:09,707 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/851c0178c787439d81cc1546c74dbd4a, entries=150, sequenceid=733, filesize=11.9 K 2023-04-05 04:59:09,707 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=114.05 KB/116790 for ae76f0bbec3b73d99b1c36f6bc76ae4f in 839ms, sequenceid=733, compaction requested=true 2023-04-05 04:59:09,707 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:59:09,707 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:09,708 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:09,708 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:09,708 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,710 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:09,710 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:09,711 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:09,711 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:09,711 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:09,713 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ae76f0bbec3b73d99b1c36f6bc76ae4f 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-05 04:59:09,713 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=A 2023-04-05 04:59:09,713 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:59:09,713 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=B 2023-04-05 04:59:09,713 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:59:09,713 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=C 2023-04-05 04:59:09,713 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-05 04:59:09,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:59:09,714 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:09,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:09,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:09,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:09,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:09,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:09,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:09,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18d5796c[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:09,715 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:09,715 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:09,716 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:09,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:09,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:09,716 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:09,716 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:09,716 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:09,716 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,716 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:09,717 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:09,717 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:09,717 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,717 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:09,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:09,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:09,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:09,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:09,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:09,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:09,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:09,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:09,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:09,720 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:09,720 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:09,720 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:09,720 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:09,720 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:09,721 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:09,721 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:09,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:09,721 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:09,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:09,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:09,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:09,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:09,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:09,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:09,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:09,722 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:09,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:09,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:09,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:09,723 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:09,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:09,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,725 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,725 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:09,725 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:09,725 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:09,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:09,725 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:09,725 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:09,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:09,725 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:09,725 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:09,725 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:09,725 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:59:09,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:09,725 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:09,725 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:09,725 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:09,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:09,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:09,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:09,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38120 deadline: 1680670809750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,825 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:09,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:09,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:09,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:09,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:09,827 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:09,827 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:09,827 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:09,827 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:09,828 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:09,828 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:09,828 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,828 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:09,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:09,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:09,829 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,829 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:09,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:09,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:09,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:09,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:09,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:09,829 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:09,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:09,830 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:09,830 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:09,830 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:09,830 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:09,830 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:09,830 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:09,830 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:09,830 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:09,830 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:09,830 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:09,830 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:09,830 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,830 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:09,830 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:09,831 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,831 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:09,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:09,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:09,831 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:09,831 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:09,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:09,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:09,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:09,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:09,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:09,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1317ff01[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:09,832 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:09,833 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:09,833 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:09,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:09,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:09,834 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:09,834 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:09,834 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:09,834 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,834 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:09,835 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:09,835 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:09,835 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:09,835 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:09,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:09,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:09,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:09,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:09,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:09,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:09,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:09,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:09,837 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:09,837 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:09,837 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:09,837 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:09,838 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:09,838 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:09,838 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:09,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:09,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:09,838 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:09,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:09,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:09,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:09,839 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:09,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:09,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:09,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:09,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:09,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:09,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:09,840 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:09,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:09,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,841 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,841 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:09,841 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:09,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:09,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:09,841 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:09,841 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,841 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:09,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:09,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:59:09,841 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:09,841 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:09,841 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:09,841 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:09,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:09,841 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:09,841 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:09,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:09,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:09,855 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:09,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38091] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38120 deadline: 1680670809854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,941 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:09,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:09,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:09,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:09,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:09,944 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:09,944 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:09,944 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:09,944 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:09,945 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:09,945 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:09,945 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:09,945 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:09,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:09,946 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,946 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:09,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:09,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:09,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:09,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:09,946 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:09,946 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:09,946 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:09,946 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:09,947 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:09,947 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:09,947 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:09,947 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:09,947 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:09,947 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:09,947 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:09,947 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:09,947 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:09,947 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:09,947 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,947 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:09,947 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:09,948 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:09,948 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:09,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:09,948 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:09,948 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:09,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:09,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:09,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:09,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:09,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:09,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25abb6a2[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:09,949 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:09,949 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:09,950 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:09,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:09,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:09,950 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:09,950 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:09,950 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,950 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:09,950 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:09,951 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,951 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:09,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:09,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:09,951 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:09,951 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:09,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:09,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:09,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:09,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:09,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:09,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:09,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:09,953 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:09,953 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:09,953 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:09,953 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:09,953 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:09,953 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:09,953 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:09,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:09,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:09,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:09,954 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:09,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:09,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:09,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:09,954 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:09,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:09,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:09,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:09,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:09,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:09,955 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:09,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:09,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,956 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,956 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:09,956 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:09,956 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:09,956 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:09,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:09,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:09,956 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:09,956 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:09,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:59:09,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:09,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:09,956 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:09,957 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:09,957 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:09,957 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:09,957 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:09,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:09,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:10,057 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:10,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:10,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:10,059 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:10,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38120 deadline: 1680670810057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:10,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:10,059 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:10,059 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:10,059 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:10,060 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:10,060 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,060 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:10,060 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,060 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:10,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:10,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:10,061 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,061 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:10,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:10,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:10,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:10,061 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:10,061 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:10,062 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:10,062 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:10,062 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:10,062 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:10,062 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:10,062 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:10,062 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:10,062 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:10,062 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:10,062 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:10,062 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:10,062 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:10,062 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,062 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:10,062 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:10,063 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,063 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:10,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:10,063 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:10,063 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:10,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:10,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:10,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:10,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:10,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bc2da71[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:10,064 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:10,064 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,065 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,065 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:10,065 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:10,065 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:10,065 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,065 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:10,066 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:10,066 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,066 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:10,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:10,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:10,066 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:10,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:10,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:10,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:10,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:10,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:10,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:10,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:10,067 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:10,068 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:10,068 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:10,068 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:10,068 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:10,068 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,068 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,068 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:10,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:10,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:10,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:10,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:10,069 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:10,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:10,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:10,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:10,070 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:10,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:10,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,072 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:10,072 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,072 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:10,072 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:10,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:10,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:10,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:10,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:59:10,072 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:10,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:10,072 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:10,072 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:10,072 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:10,072 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,072 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,072 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,072 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:10,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:10,142 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304057727c62dea794c1e849b0f25d69ef007 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304057727c62dea794c1e849b0f25d69ef007 2023-04-05 04:59:10,146 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=754, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/f436d38e8d784969b25a46df9e7f73ef 2023-04-05 04:59:10,172 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:10,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:10,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:10,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:10,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:10,175 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:10,175 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:10,175 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:10,175 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:10,175 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,175 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:10,176 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,176 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:10,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:10,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:10,176 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,176 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:10,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:10,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:10,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:10,177 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:10,177 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:10,177 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:10,177 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:10,177 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:10,177 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:10,177 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:10,177 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:10,177 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:10,177 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:10,177 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:10,177 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:10,177 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:10,177 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:10,177 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,177 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:10,178 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:10,178 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,178 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:10,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:10,179 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,179 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:10,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:10,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:10,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:10,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:10,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:10,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@786fd47c[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:10,180 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:10,180 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,180 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,181 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:10,181 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,181 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:10,181 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:10,181 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:10,181 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,181 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:10,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:10,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:10,181 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:10,182 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:10,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:10,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:10,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:10,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:10,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:10,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:10,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:10,183 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:10,183 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:10,183 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:10,183 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:10,183 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:10,184 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,184 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,184 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:10,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:10,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:10,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:10,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:10,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:10,185 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:10,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:10,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:10,186 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:10,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:10,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,187 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,187 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:10,187 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:10,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:10,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:59:10,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:10,187 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:10,187 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,187 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:10,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:10,187 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,187 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:10,187 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:10,187 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:10,187 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:10,187 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:10,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:10,287 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:10,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:10,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:10,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:10,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:10,290 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:10,290 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:10,290 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:10,290 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:10,290 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,290 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:10,291 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,291 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:10,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:10,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:10,291 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,291 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:10,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:10,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:10,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:10,292 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:10,292 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:10,292 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:10,292 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:10,292 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:10,292 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:10,292 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:10,292 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:10,292 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:10,292 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:10,293 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:10,293 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:10,293 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:10,293 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:10,293 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,293 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:10,293 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:10,293 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,293 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:10,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:10,294 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:10,294 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:10,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:10,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:10,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:10,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:10,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@542cd387[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:10,295 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:10,295 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,295 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,295 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:10,295 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:10,295 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:10,296 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,296 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:10,296 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,296 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:10,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:10,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:10,296 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:10,296 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:10,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:10,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:10,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:10,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:10,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:10,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:10,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:10,298 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:10,298 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:10,298 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:10,298 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:10,298 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:10,299 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,299 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:10,299 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:10,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:10,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:10,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:10,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:10,299 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:10,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:10,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:10,300 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:10,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:10,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,301 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,301 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:10,301 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:10,301 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:10,301 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:10,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:59:10,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:10,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:10,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:10,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:10,301 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:10,302 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:10,301 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,302 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:10,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:10,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:10,302 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,302 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,302 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,362 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-05 04:59:10,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38091] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38120 deadline: 1680670810361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,402 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:10,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:10,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:10,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:10,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:10,404 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:10,404 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:10,405 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:10,405 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:10,405 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,405 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:10,405 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,405 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:10,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:10,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:10,406 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:10,406 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:10,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:10,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:10,406 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:10,406 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:10,406 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:10,406 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:10,406 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:10,406 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:10,406 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:10,407 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:10,407 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:10,407 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:10,407 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:10,407 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:10,407 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:10,407 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:10,407 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,407 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:10,407 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:10,407 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,407 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:10,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:10,408 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,408 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:10,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:10,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:10,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:10,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:10,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:10,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cca6106[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:10,409 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:10,409 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,409 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,409 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:10,409 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:10,409 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,409 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:10,409 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:10,410 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:10,410 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,410 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:10,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:10,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:10,410 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:10,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:10,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:10,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:10,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:10,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:10,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:10,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:10,411 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:10,411 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:10,411 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:10,412 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:10,412 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:10,412 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,412 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:10,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:10,412 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:10,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,413 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:10,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:10,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:10,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:10,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:10,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:10,414 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:10,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:10,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,415 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:10,415 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,415 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:10,415 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:10,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:59:10,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:10,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:10,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:10,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:10,415 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:10,415 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:10,415 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,415 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,415 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,415 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:10,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:10,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:10,415 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,415 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:10,516 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:10,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:10,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:10,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:10,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:10,518 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:10,518 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:10,518 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:10,518 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:10,518 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,519 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:10,519 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,519 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:10,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:10,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:10,519 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,519 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:10,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:10,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:10,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:10,520 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:10,520 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:10,520 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:10,520 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:10,520 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:10,521 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:10,521 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:10,521 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:10,521 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:10,521 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:10,521 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:10,521 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:10,521 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:10,521 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,521 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:10,521 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:10,522 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,522 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:10,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:10,523 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,523 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:10,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:10,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:10,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:10,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:10,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:10,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@718637bc[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:10,524 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:10,524 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,524 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,524 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:10,524 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:10,524 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:10,525 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,525 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:10,525 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,525 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:10,525 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:10,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:10,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:10,525 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:10,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:10,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:10,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:10,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:10,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:10,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:10,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:10,527 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:10,527 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:10,527 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:10,527 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:10,527 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:10,528 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,528 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:10,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:10,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,528 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:10,528 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:10,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:10,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:10,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:10,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:10,529 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:10,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:10,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:10,531 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,531 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:10,531 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:10,531 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,531 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,531 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:10,531 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:10,531 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:10,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:10,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:10,531 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:10,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-05 04:59:10,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:10,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:10,531 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,531 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:10,531 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:10,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:10,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:10,558 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=754 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/2756b81e1eb747e2a7e43e6e9f8cb299 2023-04-05 04:59:10,572 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=754 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/c4183975f76043daa3a2113b79468b37 2023-04-05 04:59:10,575 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/f436d38e8d784969b25a46df9e7f73ef as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/f436d38e8d784969b25a46df9e7f73ef 2023-04-05 04:59:10,587 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/f436d38e8d784969b25a46df9e7f73ef, entries=150, sequenceid=754, filesize=25.4 K 2023-04-05 04:59:10,588 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/2756b81e1eb747e2a7e43e6e9f8cb299 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/2756b81e1eb747e2a7e43e6e9f8cb299 2023-04-05 04:59:10,590 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/2756b81e1eb747e2a7e43e6e9f8cb299, entries=150, sequenceid=754, filesize=11.9 K 2023-04-05 04:59:10,591 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/c4183975f76043daa3a2113b79468b37 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/c4183975f76043daa3a2113b79468b37 2023-04-05 04:59:10,594 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/c4183975f76043daa3a2113b79468b37, entries=150, sequenceid=754, filesize=11.9 K 2023-04-05 04:59:10,594 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for ae76f0bbec3b73d99b1c36f6bc76ae4f in 881ms, sequenceid=754, compaction requested=true 2023-04-05 04:59:10,594 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:59:10,594 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:59:10,594 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:59:10,595 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:59:10,595 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:59:10,595 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:59:10,595 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111838 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:59:10,596 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1912): ae76f0bbec3b73d99b1c36f6bc76ae4f/A is initiating minor compaction (all files) 2023-04-05 04:59:10,596 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:59:10,596 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ae76f0bbec3b73d99b1c36f6bc76ae4f/A in TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:10,596 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): ae76f0bbec3b73d99b1c36f6bc76ae4f/B is initiating minor compaction (all files) 2023-04-05 04:59:10,596 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/2d3bcaabd6f74b72a2ba7d57dce1f5a3, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/52681e1a32dd4dbea38ed0949d18edc0, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/addfbf1037784c1cad2ef311c39e775f, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/f436d38e8d784969b25a46df9e7f73ef] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp, totalSize=109.2 K 2023-04-05 04:59:10,596 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ae76f0bbec3b73d99b1c36f6bc76ae4f/B in TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:10,596 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/307b20ef9ca14ac592ac9694d666b5b5, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/25a4cce421d34a3894e2ed9da46e3e69, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/fcc2ceee291b472291f6a3f0d5857888, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/2756b81e1eb747e2a7e43e6e9f8cb299] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp, totalSize=48.2 K 2023-04-05 04:59:10,596 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 307b20ef9ca14ac592ac9694d666b5b5, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=684, earliestPutTs=1680670744028 2023-04-05 04:59:10,596 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 2d3bcaabd6f74b72a2ba7d57dce1f5a3, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=684, earliestPutTs=1680670744028 2023-04-05 04:59:10,596 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 25a4cce421d34a3894e2ed9da46e3e69, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=721, earliestPutTs=1680670746358 2023-04-05 04:59:10,596 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting 52681e1a32dd4dbea38ed0949d18edc0, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=721, earliestPutTs=1680670746358 2023-04-05 04:59:10,597 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting fcc2ceee291b472291f6a3f0d5857888, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=733, earliestPutTs=1680670747515 2023-04-05 04:59:10,597 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting addfbf1037784c1cad2ef311c39e775f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=733, earliestPutTs=1680670747515 2023-04-05 04:59:10,597 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 2756b81e1eb747e2a7e43e6e9f8cb299, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=754, earliestPutTs=1680670749688 2023-04-05 04:59:10,597 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] compactions.Compactor(207): Compacting f436d38e8d784969b25a46df9e7f73ef, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=754, earliestPutTs=1680670749688 2023-04-05 04:59:10,604 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): ae76f0bbec3b73d99b1c36f6bc76ae4f#B#compaction#525 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:59:10,612 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] throttle.PressureAwareThroughputController(145): ae76f0bbec3b73d99b1c36f6bc76ae4f#A#compaction#524 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:59:10,621 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/bf2c81ca75004eec85d42fedd6b97be1 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/bf2c81ca75004eec85d42fedd6b97be1 2023-04-05 04:59:10,631 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:10,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:10,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-05 04:59:10,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-05 04:59:10,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-05 04:59:10,634 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-05 04:59:10,634 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:10,634 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-05 04:59:10,634 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-05 04:59:10,634 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,634 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:10,635 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:10,635 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:10,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:10,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:10,636 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:10,636 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ae76f0bbec3b73d99b1c36f6bc76ae4f/B of ae76f0bbec3b73d99b1c36f6bc76ae4f into bf2c81ca75004eec85d42fedd6b97be1(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:59:10,636 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-05 04:59:10,636 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:59:10,636 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f., storeName=ae76f0bbec3b73d99b1c36f6bc76ae4f/B, priority=12, startTime=1680670750595; duration=0sec 2023-04-05 04:59:10,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:10,636 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-05 04:59:10,636 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-05 04:59:10,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-05 04:59:10,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:10,636 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-05 04:59:10,637 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-05 04:59:10,637 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-05 04:59:10,637 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-05 04:59:10,637 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-05 04:59:10,637 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-05 04:59:10,637 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-05 04:59:10,637 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-05 04:59:10,637 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:10,637 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-05 04:59:10,637 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. started... 2023-04-05 04:59:10,637 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1912): ae76f0bbec3b73d99b1c36f6bc76ae4f/C is initiating minor compaction (all files) 2023-04-05 04:59:10,637 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ae76f0bbec3b73d99b1c36f6bc76ae4f 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-05 04:59:10,637 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ae76f0bbec3b73d99b1c36f6bc76ae4f/C in TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:10,637 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/e1108cf49bb24ca1b42441ce97313b26, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/267117be1c164a4db29cf25dd4ca3b3d, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/851c0178c787439d81cc1546c74dbd4a, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/c4183975f76043daa3a2113b79468b37] into tmpdir=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp, totalSize=48.2 K 2023-04-05 04:59:10,637 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=A 2023-04-05 04:59:10,638 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:59:10,638 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=B 2023-04-05 04:59:10,638 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:59:10,638 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=C 2023-04-05 04:59:10,638 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting e1108cf49bb24ca1b42441ce97313b26, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=684, earliestPutTs=1680670744028 2023-04-05 04:59:10,638 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:59:10,638 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 267117be1c164a4db29cf25dd4ca3b3d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=721, earliestPutTs=1680670746358 2023-04-05 04:59:10,638 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting 851c0178c787439d81cc1546c74dbd4a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=733, earliestPutTs=1680670747515 2023-04-05 04:59:10,639 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] compactions.Compactor(207): Compacting c4183975f76043daa3a2113b79468b37, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=754, earliestPutTs=1680670749688 2023-04-05 04:59:10,652 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] throttle.PressureAwareThroughputController(145): ae76f0bbec3b73d99b1c36f6bc76ae4f#C#compaction#526 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-05 04:59:10,668 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/ea24dc769e5949bb96f2f9557c53bb33 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/ea24dc769e5949bb96f2f9557c53bb33 2023-04-05 04:59:10,671 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230405cc14104005a549aea13f478b2444fe9b to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405cc14104005a549aea13f478b2444fe9b 2023-04-05 04:59:10,672 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ae76f0bbec3b73d99b1c36f6bc76ae4f/A of ae76f0bbec3b73d99b1c36f6bc76ae4f into ea24dc769e5949bb96f2f9557c53bb33(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:59:10,672 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:59:10,672 INFO [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f., storeName=ae76f0bbec3b73d99b1c36f6bc76ae4f/A, priority=12, startTime=1680670750594; duration=0sec 2023-04-05 04:59:10,672 DEBUG [RS:0;jenkins-hbase20:38091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:59:10,675 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=766, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/818ed97a6cbe45b08dfb0024861ee1c8 2023-04-05 04:59:10,681 DEBUG [Thread-3544] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7689435c to 127.0.0.1:56229 2023-04-05 04:59:10,681 DEBUG [Thread-3544] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:59:10,682 DEBUG [Thread-3538] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49c34fbe to 127.0.0.1:56229 2023-04-05 04:59:10,682 DEBUG [Thread-3538] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:59:10,682 DEBUG [Thread-3546] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x475ee808 to 127.0.0.1:56229 2023-04-05 04:59:10,682 DEBUG [Thread-3546] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:59:10,683 DEBUG [Thread-3542] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x44535519 to 127.0.0.1:56229 2023-04-05 04:59:10,683 DEBUG [Thread-3542] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:59:10,683 DEBUG [Thread-3540] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5467be78 to 127.0.0.1:56229 2023-04-05 04:59:10,683 DEBUG [Thread-3540] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:59:10,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38091] regionserver.HRegion(9158): Flush requested on ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:59:10,869 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. as already flushing 2023-04-05 04:59:10,870 DEBUG [Thread-3535] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34b8bf14 to 127.0.0.1:56229 2023-04-05 04:59:10,870 DEBUG [Thread-3535] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:59:11,068 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/d9b39bbca1e04f3e88d0238370db7d66 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/d9b39bbca1e04f3e88d0238370db7d66 2023-04-05 04:59:11,073 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ae76f0bbec3b73d99b1c36f6bc76ae4f/C of ae76f0bbec3b73d99b1c36f6bc76ae4f into d9b39bbca1e04f3e88d0238370db7d66(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-05 04:59:11,073 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:59:11,073 INFO [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f., storeName=ae76f0bbec3b73d99b1c36f6bc76ae4f/C, priority=12, startTime=1680670750595; duration=0sec 2023-04-05 04:59:11,073 DEBUG [RS:0;jenkins-hbase20:38091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-05 04:59:11,088 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=766 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/d3203a1cca5e4afbaec0357f7119fbce 2023-04-05 04:59:11,502 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=766 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/8091f3a0554247d49cfad8e2fb3cccec 2023-04-05 04:59:11,511 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/818ed97a6cbe45b08dfb0024861ee1c8 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/818ed97a6cbe45b08dfb0024861ee1c8 2023-04-05 04:59:11,516 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/818ed97a6cbe45b08dfb0024861ee1c8, entries=150, sequenceid=766, filesize=25.4 K 2023-04-05 04:59:11,517 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/d3203a1cca5e4afbaec0357f7119fbce as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/d3203a1cca5e4afbaec0357f7119fbce 2023-04-05 04:59:11,523 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/d3203a1cca5e4afbaec0357f7119fbce, entries=150, sequenceid=766, filesize=11.9 K 2023-04-05 04:59:11,524 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/8091f3a0554247d49cfad8e2fb3cccec as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/8091f3a0554247d49cfad8e2fb3cccec 2023-04-05 04:59:11,527 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/8091f3a0554247d49cfad8e2fb3cccec, entries=150, sequenceid=766, filesize=11.9 K 2023-04-05 04:59:11,528 INFO [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=6.71 KB/6870 for ae76f0bbec3b73d99b1c36f6bc76ae4f in 891ms, sequenceid=766, compaction requested=false 2023-04-05 04:59:11,528 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:59:11,528 DEBUG [rs(jenkins-hbase20.apache.org,38091,1680670569206)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:11,528 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-05 04:59:11,528 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-05 04:59:11,528 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:11,528 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-05 04:59:11,528 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-05 04:59:11,529 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:11,529 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:11,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:11,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:11,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:11,530 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:11,530 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-05 04:59:11,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:11,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:11,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:11,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:11,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:11,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,38091,1680670569206' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-05 04:59:11,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77ccd1c4[Count = 0] remaining members to acquire global barrier 2023-04-05 04:59:11,532 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-05 04:59:11,532 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:11,532 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:11,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:11,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:11,533 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-05 04:59:11,533 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-05 04:59:11,533 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,38091,1680670569206' in zk 2023-04-05 04:59:11,533 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:11,533 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-05 04:59:11,533 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-05 04:59:11,533 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:11,533 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:11,533 DEBUG [member: 'jenkins-hbase20.apache.org,38091,1680670569206' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-05 04:59:11,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:11,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:11,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:11,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:11,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:11,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:11,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:11,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:11,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:11,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:11,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,38091,1680670569206': 2023-04-05 04:59:11,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,38091,1680670569206' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-05 04:59:11,537 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-05 04:59:11,537 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-05 04:59:11,537 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-05 04:59:11,537 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-05 04:59:11,537 INFO [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-05 04:59:11,541 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:11,541 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:11,541 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:11,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:11,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:11,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-05 04:59:11,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-05 04:59:11,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:11,542 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:11,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-05 04:59:11,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:11,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:11,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:11,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:11,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-05 04:59:11,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:11,543 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:11,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:11,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-05 04:59:11,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-05 04:59:11,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:11,545 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:11,545 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-05 04:59:11,545 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-05 04:59:11,545 DEBUG [(jenkins-hbase20.apache.org,43797,1680670568131)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-05 04:59:11,545 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:11,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-05 04:59:11,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-05 04:59:11,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-05 04:59:11,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-05 04:59:11,545 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-05 04:59:11,546 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:11,546 DEBUG [Thread-3537] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-05 04:59:11,546 DEBUG [Thread-3537] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-05 04:59:11,546 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:11,546 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-05 04:59:11,546 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-05 04:59:11,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-05 04:59:11,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-05 04:59:11,646 DEBUG [Thread-3537] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-05 04:59:11,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-05 04:59:14,690 DEBUG [Thread-3533] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7bec0400 to 127.0.0.1:56229 2023-04-05 04:59:14,690 DEBUG [Thread-3533] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:59:17,819 DEBUG [Thread-3527] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x630ea023 to 127.0.0.1:56229 2023-04-05 04:59:17,819 DEBUG [Thread-3527] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:59:17,835 DEBUG [Thread-3529] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ec98d63 to 127.0.0.1:56229 2023-04-05 04:59:17,835 DEBUG [Thread-3529] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:59:17,872 DEBUG [Thread-3531] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1ae3ec16 to 127.0.0.1:56229 2023-04-05 04:59:17,872 DEBUG [Thread-3531] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:59:17,872 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-05 04:59:17,873 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(394): wrote 99 2023-04-05 04:59:17,873 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(394): wrote 100 2023-04-05 04:59:17,873 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(394): wrote 142 2023-04-05 04:59:17,873 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(394): wrote 122 2023-04-05 04:59:17,873 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(394): wrote 232 2023-04-05 04:59:17,873 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-05 04:59:17,874 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(398): read 5497 2023-04-05 04:59:17,874 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(398): read 5850 2023-04-05 04:59:17,874 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(398): read 5695 2023-04-05 04:59:17,874 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(398): read 5535 2023-04-05 04:59:17,874 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(398): read 5834 2023-04-05 04:59:17,874 INFO [Listener at localhost.localdomain/34579] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-05 04:59:17,874 INFO [Listener at localhost.localdomain/34579] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-05 04:59:17,874 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ddb3e63 to 127.0.0.1:56229 2023-04-05 04:59:17,875 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:59:17,876 INFO [Listener at localhost.localdomain/34579] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-05 04:59:17,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-05 04:59:17,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-05 04:59:17,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-05 04:59:17,882 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680670757882"}]},"ts":"1680670757882"} 2023-04-05 04:59:17,883 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-05 04:59:17,891 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-05 04:59:17,891 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ae76f0bbec3b73d99b1c36f6bc76ae4f, UNASSIGN}] 2023-04-05 04:59:17,892 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ae76f0bbec3b73d99b1c36f6bc76ae4f, UNASSIGN 2023-04-05 04:59:17,893 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=ae76f0bbec3b73d99b1c36f6bc76ae4f, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:17,893 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670757893"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680670757893"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680670757893"}]},"ts":"1680670757893"} 2023-04-05 04:59:17,894 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206}] 2023-04-05 04:59:17,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-05 04:59:18,046 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:59:18,047 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing ae76f0bbec3b73d99b1c36f6bc76ae4f, disabling compactions & flushes 2023-04-05 04:59:18,047 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:18,047 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:18,047 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. after waiting 0 ms 2023-04-05 04:59:18,047 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:18,047 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing ae76f0bbec3b73d99b1c36f6bc76ae4f 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-05 04:59:18,047 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=A 2023-04-05 04:59:18,047 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:59:18,047 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=B 2023-04-05 04:59:18,047 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:59:18,047 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ae76f0bbec3b73d99b1c36f6bc76ae4f, store=C 2023-04-05 04:59:18,047 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-05 04:59:18,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-05 04:59:18,463 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304054491de5276e0441f8affbc4c6bc5dcdc to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304054491de5276e0441f8affbc4c6bc5dcdc 2023-04-05 04:59:18,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-05 04:59:18,870 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=776, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/e5ab89e4cd8c4b07b6019b218ef2c611 2023-04-05 04:59:18,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-05 04:59:19,286 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=776 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/5371eb87526341bcb92abd1484266ca1 2023-04-05 04:59:19,303 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=776 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/5b1e9a1a472440b180715de3dfe7759d 2023-04-05 04:59:19,308 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/A/e5ab89e4cd8c4b07b6019b218ef2c611 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/e5ab89e4cd8c4b07b6019b218ef2c611 2023-04-05 04:59:19,313 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/e5ab89e4cd8c4b07b6019b218ef2c611, entries=150, sequenceid=776, filesize=25.4 K 2023-04-05 04:59:19,314 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/B/5371eb87526341bcb92abd1484266ca1 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/5371eb87526341bcb92abd1484266ca1 2023-04-05 04:59:19,321 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/5371eb87526341bcb92abd1484266ca1, entries=150, sequenceid=776, filesize=11.9 K 2023-04-05 04:59:19,322 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/.tmp/C/5b1e9a1a472440b180715de3dfe7759d as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/5b1e9a1a472440b180715de3dfe7759d 2023-04-05 04:59:19,327 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/5b1e9a1a472440b180715de3dfe7759d, entries=150, sequenceid=776, filesize=11.9 K 2023-04-05 04:59:19,328 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for ae76f0bbec3b73d99b1c36f6bc76ae4f in 1281ms, sequenceid=776, compaction requested=true 2023-04-05 04:59:19,348 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/d5ec479c08424a7b9783c4833718aaf7, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/1c9cc481f75f4df48f7ad6a75cd2fc0c, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/4f27228c411a4f1b972624702521872e, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/d54123f6f469412d92b670130d3f308e, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/c8d2beff817842d69852c73b72c36924, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/3740d85fee78431a96885921cc09bfee, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/76df34c5008643958b0b07d3f4a214ef, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/3ca36fdff31f4b948c8436c5207a560c, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/7a92851f34504a2495e92ba4ff7039fe, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/96e47bfa486b47a6b04e84eb1637e08a, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/e824346f0be34c4fa519b14295be47d9, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/68835a4d706b4c1695628647066a00ee, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/5791a60dc4ad4e9dacf4e113a4927b98, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/9157d49340e64eb1b7d6d7bece16491a, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/de651601efbc4d62b58a98f9b899516d, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/ff49ad34719c4184812632266d9e3136, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/cf6858a296a449b5a27cca01ce97fbb4, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/e91b81194ae14dac94eb8a31954597dc, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/2d3bcaabd6f74b72a2ba7d57dce1f5a3, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/52681e1a32dd4dbea38ed0949d18edc0, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/addfbf1037784c1cad2ef311c39e775f, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/f436d38e8d784969b25a46df9e7f73ef] to archive 2023-04-05 04:59:19,349 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 04:59:19,350 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/d5ec479c08424a7b9783c4833718aaf7 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/d5ec479c08424a7b9783c4833718aaf7 2023-04-05 04:59:19,351 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/1c9cc481f75f4df48f7ad6a75cd2fc0c to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/1c9cc481f75f4df48f7ad6a75cd2fc0c 2023-04-05 04:59:19,352 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/4f27228c411a4f1b972624702521872e to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/4f27228c411a4f1b972624702521872e 2023-04-05 04:59:19,352 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/d54123f6f469412d92b670130d3f308e to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/d54123f6f469412d92b670130d3f308e 2023-04-05 04:59:19,353 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/c8d2beff817842d69852c73b72c36924 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/c8d2beff817842d69852c73b72c36924 2023-04-05 04:59:19,354 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/3740d85fee78431a96885921cc09bfee to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/3740d85fee78431a96885921cc09bfee 2023-04-05 04:59:19,354 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/76df34c5008643958b0b07d3f4a214ef to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/76df34c5008643958b0b07d3f4a214ef 2023-04-05 04:59:19,355 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/3ca36fdff31f4b948c8436c5207a560c to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/3ca36fdff31f4b948c8436c5207a560c 2023-04-05 04:59:19,356 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/7a92851f34504a2495e92ba4ff7039fe to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/7a92851f34504a2495e92ba4ff7039fe 2023-04-05 04:59:19,357 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/96e47bfa486b47a6b04e84eb1637e08a to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/96e47bfa486b47a6b04e84eb1637e08a 2023-04-05 04:59:19,358 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/e824346f0be34c4fa519b14295be47d9 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/e824346f0be34c4fa519b14295be47d9 2023-04-05 04:59:19,359 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/68835a4d706b4c1695628647066a00ee to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/68835a4d706b4c1695628647066a00ee 2023-04-05 04:59:19,360 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/5791a60dc4ad4e9dacf4e113a4927b98 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/5791a60dc4ad4e9dacf4e113a4927b98 2023-04-05 04:59:19,361 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/9157d49340e64eb1b7d6d7bece16491a to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/9157d49340e64eb1b7d6d7bece16491a 2023-04-05 04:59:19,362 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/de651601efbc4d62b58a98f9b899516d to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/de651601efbc4d62b58a98f9b899516d 2023-04-05 04:59:19,363 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/ff49ad34719c4184812632266d9e3136 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/ff49ad34719c4184812632266d9e3136 2023-04-05 04:59:19,364 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/cf6858a296a449b5a27cca01ce97fbb4 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/cf6858a296a449b5a27cca01ce97fbb4 2023-04-05 04:59:19,365 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/e91b81194ae14dac94eb8a31954597dc to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/e91b81194ae14dac94eb8a31954597dc 2023-04-05 04:59:19,366 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/2d3bcaabd6f74b72a2ba7d57dce1f5a3 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/2d3bcaabd6f74b72a2ba7d57dce1f5a3 2023-04-05 04:59:19,366 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/52681e1a32dd4dbea38ed0949d18edc0 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/52681e1a32dd4dbea38ed0949d18edc0 2023-04-05 04:59:19,367 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/addfbf1037784c1cad2ef311c39e775f to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/addfbf1037784c1cad2ef311c39e775f 2023-04-05 04:59:19,368 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/f436d38e8d784969b25a46df9e7f73ef to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/f436d38e8d784969b25a46df9e7f73ef 2023-04-05 04:59:19,371 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/c9b802fd70b14ef5b9ff33b9fe31ebfb, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/62c669d8076a43ab831e658745683c4b, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/ff9381ebcd8749cd8e07ab5d55e75da9, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/33107c9dcd8f4354b5db6916d9f0e883, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/dc85a1c74c524df1be40a21610aba247, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/56c1eb9505cd4d35b745bb80b6f1e5a2, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/8b6d1e6220cd433a9b51858476ab2c47, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/68005163ce124a04b7d9d2ba3f360cb0, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/335531c356a040a4ae376e7039d56e91, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/768c871f9c9744efa06646a3a7f40d6e, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/cb090e9a856949789a6fb0e640607e7b, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/4349c68abb9b460384fe3414103c621e, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/4920cf1c7a04433fbffdb6c98d5a66b2, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/a37ae1a2edf04942a6c36b164c25a762, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/472ac4f9f09f4708809ccb832e86328e, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/4af017fb1b134c458f672d4e68c562af, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/a631481bb6d748d580037f2ce466cdd7, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/307b20ef9ca14ac592ac9694d666b5b5, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/10bbafa6ddbf492583fdf0dab26b2e5f, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/25a4cce421d34a3894e2ed9da46e3e69, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/fcc2ceee291b472291f6a3f0d5857888, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/2756b81e1eb747e2a7e43e6e9f8cb299] to archive 2023-04-05 04:59:19,371 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 04:59:19,372 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/c9b802fd70b14ef5b9ff33b9fe31ebfb to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/c9b802fd70b14ef5b9ff33b9fe31ebfb 2023-04-05 04:59:19,373 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/62c669d8076a43ab831e658745683c4b to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/62c669d8076a43ab831e658745683c4b 2023-04-05 04:59:19,374 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/ff9381ebcd8749cd8e07ab5d55e75da9 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/ff9381ebcd8749cd8e07ab5d55e75da9 2023-04-05 04:59:19,375 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/33107c9dcd8f4354b5db6916d9f0e883 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/33107c9dcd8f4354b5db6916d9f0e883 2023-04-05 04:59:19,375 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/dc85a1c74c524df1be40a21610aba247 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/dc85a1c74c524df1be40a21610aba247 2023-04-05 04:59:19,376 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/56c1eb9505cd4d35b745bb80b6f1e5a2 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/56c1eb9505cd4d35b745bb80b6f1e5a2 2023-04-05 04:59:19,377 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/8b6d1e6220cd433a9b51858476ab2c47 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/8b6d1e6220cd433a9b51858476ab2c47 2023-04-05 04:59:19,377 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/68005163ce124a04b7d9d2ba3f360cb0 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/68005163ce124a04b7d9d2ba3f360cb0 2023-04-05 04:59:19,378 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/335531c356a040a4ae376e7039d56e91 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/335531c356a040a4ae376e7039d56e91 2023-04-05 04:59:19,379 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/768c871f9c9744efa06646a3a7f40d6e to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/768c871f9c9744efa06646a3a7f40d6e 2023-04-05 04:59:19,379 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/cb090e9a856949789a6fb0e640607e7b to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/cb090e9a856949789a6fb0e640607e7b 2023-04-05 04:59:19,380 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/4349c68abb9b460384fe3414103c621e to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/4349c68abb9b460384fe3414103c621e 2023-04-05 04:59:19,381 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/4920cf1c7a04433fbffdb6c98d5a66b2 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/4920cf1c7a04433fbffdb6c98d5a66b2 2023-04-05 04:59:19,381 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/a37ae1a2edf04942a6c36b164c25a762 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/a37ae1a2edf04942a6c36b164c25a762 2023-04-05 04:59:19,382 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/472ac4f9f09f4708809ccb832e86328e to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/472ac4f9f09f4708809ccb832e86328e 2023-04-05 04:59:19,383 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/4af017fb1b134c458f672d4e68c562af to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/4af017fb1b134c458f672d4e68c562af 2023-04-05 04:59:19,384 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/a631481bb6d748d580037f2ce466cdd7 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/a631481bb6d748d580037f2ce466cdd7 2023-04-05 04:59:19,384 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/307b20ef9ca14ac592ac9694d666b5b5 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/307b20ef9ca14ac592ac9694d666b5b5 2023-04-05 04:59:19,385 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/10bbafa6ddbf492583fdf0dab26b2e5f to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/10bbafa6ddbf492583fdf0dab26b2e5f 2023-04-05 04:59:19,385 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/25a4cce421d34a3894e2ed9da46e3e69 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/25a4cce421d34a3894e2ed9da46e3e69 2023-04-05 04:59:19,386 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/fcc2ceee291b472291f6a3f0d5857888 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/fcc2ceee291b472291f6a3f0d5857888 2023-04-05 04:59:19,387 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/2756b81e1eb747e2a7e43e6e9f8cb299 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/2756b81e1eb747e2a7e43e6e9f8cb299 2023-04-05 04:59:19,390 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/c32fca2ae9eb49a99a7cc2e9d251ebbd, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/6c45d31e0a9040189de6c478d88c3448, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/701b0d86095b4c94ae1b322cce572f98, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/14c954960e6140adb6f1c7b94c0f9924, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/f517064e4a4a43f68bd5f19c1c1ccea9, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/86d51637b14743999dd9800e3792830a, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/e513e2369c194d1babad81a3c9900670, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/4222d5bf2a014f9aada4ddf58b28d4ba, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/700a9f652732416db63f544a0eb6408d, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/a1f3106e10ad4d25b3cdadc93e2905e7, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/8b0b2565269149419153b2254e39c1e6, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/19e1d8f3155d4e96a5b8fd7545329a17, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/bd94f1d302e6433fb430c2736c86e9ff, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/3f244370122e40e6aaf384de4ad6e8b2, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/73036436b39846348760b41a086346d9, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/bcf0e035cc214af0afaa5723e4c66141, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/3d045cb70a814a67bc06cdf38c4113a6, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/e1108cf49bb24ca1b42441ce97313b26, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/2a171f4627634618951c647eb9e2039e, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/267117be1c164a4db29cf25dd4ca3b3d, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/851c0178c787439d81cc1546c74dbd4a, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/c4183975f76043daa3a2113b79468b37] to archive 2023-04-05 04:59:19,390 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-05 04:59:19,391 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/c32fca2ae9eb49a99a7cc2e9d251ebbd to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/c32fca2ae9eb49a99a7cc2e9d251ebbd 2023-04-05 04:59:19,392 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/6c45d31e0a9040189de6c478d88c3448 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/6c45d31e0a9040189de6c478d88c3448 2023-04-05 04:59:19,393 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/701b0d86095b4c94ae1b322cce572f98 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/701b0d86095b4c94ae1b322cce572f98 2023-04-05 04:59:19,394 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/14c954960e6140adb6f1c7b94c0f9924 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/14c954960e6140adb6f1c7b94c0f9924 2023-04-05 04:59:19,395 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/f517064e4a4a43f68bd5f19c1c1ccea9 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/f517064e4a4a43f68bd5f19c1c1ccea9 2023-04-05 04:59:19,396 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/86d51637b14743999dd9800e3792830a to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/86d51637b14743999dd9800e3792830a 2023-04-05 04:59:19,397 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/e513e2369c194d1babad81a3c9900670 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/e513e2369c194d1babad81a3c9900670 2023-04-05 04:59:19,398 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/4222d5bf2a014f9aada4ddf58b28d4ba to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/4222d5bf2a014f9aada4ddf58b28d4ba 2023-04-05 04:59:19,399 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/700a9f652732416db63f544a0eb6408d to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/700a9f652732416db63f544a0eb6408d 2023-04-05 04:59:19,400 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/a1f3106e10ad4d25b3cdadc93e2905e7 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/a1f3106e10ad4d25b3cdadc93e2905e7 2023-04-05 04:59:19,402 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/8b0b2565269149419153b2254e39c1e6 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/8b0b2565269149419153b2254e39c1e6 2023-04-05 04:59:19,403 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/19e1d8f3155d4e96a5b8fd7545329a17 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/19e1d8f3155d4e96a5b8fd7545329a17 2023-04-05 04:59:19,404 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/bd94f1d302e6433fb430c2736c86e9ff to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/bd94f1d302e6433fb430c2736c86e9ff 2023-04-05 04:59:19,405 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/3f244370122e40e6aaf384de4ad6e8b2 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/3f244370122e40e6aaf384de4ad6e8b2 2023-04-05 04:59:19,406 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/73036436b39846348760b41a086346d9 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/73036436b39846348760b41a086346d9 2023-04-05 04:59:19,407 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/bcf0e035cc214af0afaa5723e4c66141 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/bcf0e035cc214af0afaa5723e4c66141 2023-04-05 04:59:19,408 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/3d045cb70a814a67bc06cdf38c4113a6 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/3d045cb70a814a67bc06cdf38c4113a6 2023-04-05 04:59:19,409 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/e1108cf49bb24ca1b42441ce97313b26 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/e1108cf49bb24ca1b42441ce97313b26 2023-04-05 04:59:19,410 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/2a171f4627634618951c647eb9e2039e to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/2a171f4627634618951c647eb9e2039e 2023-04-05 04:59:19,411 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/267117be1c164a4db29cf25dd4ca3b3d to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/267117be1c164a4db29cf25dd4ca3b3d 2023-04-05 04:59:19,413 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/851c0178c787439d81cc1546c74dbd4a to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/851c0178c787439d81cc1546c74dbd4a 2023-04-05 04:59:19,414 DEBUG [StoreCloser-TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/c4183975f76043daa3a2113b79468b37 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/c4183975f76043daa3a2113b79468b37 2023-04-05 04:59:19,418 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/recovered.edits/779.seqid, newMaxSeqId=779, maxSeqId=4 2023-04-05 04:59:19,420 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f. 2023-04-05 04:59:19,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for ae76f0bbec3b73d99b1c36f6bc76ae4f: 2023-04-05 04:59:19,422 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:59:19,423 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=ae76f0bbec3b73d99b1c36f6bc76ae4f, regionState=CLOSED 2023-04-05 04:59:19,423 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680670759423"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680670759423"}]},"ts":"1680670759423"} 2023-04-05 04:59:19,425 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-05 04:59:19,426 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure ae76f0bbec3b73d99b1c36f6bc76ae4f, server=jenkins-hbase20.apache.org,38091,1680670569206 in 1.5300 sec 2023-04-05 04:59:19,427 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-05 04:59:19,427 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=ae76f0bbec3b73d99b1c36f6bc76ae4f, UNASSIGN in 1.5350 sec 2023-04-05 04:59:19,428 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680670759428"}]},"ts":"1680670759428"} 2023-04-05 04:59:19,429 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-05 04:59:19,431 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-05 04:59:19,433 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5550 sec 2023-04-05 04:59:19,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-05 04:59:19,992 INFO [Listener at localhost.localdomain/34579] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-05 04:59:19,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-05 04:59:19,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 04:59:19,996 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 04:59:19,997 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 04:59:19,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-05 04:59:20,004 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:59:20,008 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A, FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B, FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C, FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/recovered.edits] 2023-04-05 04:59:20,013 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/818ed97a6cbe45b08dfb0024861ee1c8 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/818ed97a6cbe45b08dfb0024861ee1c8 2023-04-05 04:59:20,016 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/e5ab89e4cd8c4b07b6019b218ef2c611 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/e5ab89e4cd8c4b07b6019b218ef2c611 2023-04-05 04:59:20,018 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/ea24dc769e5949bb96f2f9557c53bb33 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/A/ea24dc769e5949bb96f2f9557c53bb33 2023-04-05 04:59:20,024 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/5371eb87526341bcb92abd1484266ca1 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/5371eb87526341bcb92abd1484266ca1 2023-04-05 04:59:20,026 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/bf2c81ca75004eec85d42fedd6b97be1 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/bf2c81ca75004eec85d42fedd6b97be1 2023-04-05 04:59:20,028 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/d3203a1cca5e4afbaec0357f7119fbce to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/B/d3203a1cca5e4afbaec0357f7119fbce 2023-04-05 04:59:20,033 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/5b1e9a1a472440b180715de3dfe7759d to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/5b1e9a1a472440b180715de3dfe7759d 2023-04-05 04:59:20,035 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/8091f3a0554247d49cfad8e2fb3cccec to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/8091f3a0554247d49cfad8e2fb3cccec 2023-04-05 04:59:20,037 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/d9b39bbca1e04f3e88d0238370db7d66 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/C/d9b39bbca1e04f3e88d0238370db7d66 2023-04-05 04:59:20,042 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/recovered.edits/779.seqid to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f/recovered.edits/779.seqid 2023-04-05 04:59:20,043 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/.tmp/data/default/TestAcidGuarantees/ae76f0bbec3b73d99b1c36f6bc76ae4f 2023-04-05 04:59:20,043 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-05 04:59:20,044 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-05 04:59:20,045 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-05 04:59:20,048 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304051867afe2980f4eac997831fa6a4db23e to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304051867afe2980f4eac997831fa6a4db23e 2023-04-05 04:59:20,049 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304051ce339ad7f144ad296970ec8b5efb9b8 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304051ce339ad7f144ad296970ec8b5efb9b8 2023-04-05 04:59:20,051 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304052e9493d1b76845ce9820b3b6ec859c4d to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304052e9493d1b76845ce9820b3b6ec859c4d 2023-04-05 04:59:20,052 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040539c092314046487a825b8e81ec16668d to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040539c092314046487a825b8e81ec16668d 2023-04-05 04:59:20,053 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304053b2978d8050e4c6481f66750c2812ed4 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304053b2978d8050e4c6481f66750c2812ed4 2023-04-05 04:59:20,054 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304054491de5276e0441f8affbc4c6bc5dcdc to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304054491de5276e0441f8affbc4c6bc5dcdc 2023-04-05 04:59:20,055 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304055d42d5f6847d4dd18ec64d4ad876c049 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304055d42d5f6847d4dd18ec64d4ad876c049 2023-04-05 04:59:20,056 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304055dfae42ca5f643929e077d231d494ce9 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304055dfae42ca5f643929e077d231d494ce9 2023-04-05 04:59:20,057 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405661e0c5a542743c1b62c11f0db2c0804 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405661e0c5a542743c1b62c11f0db2c0804 2023-04-05 04:59:20,059 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304057727c62dea794c1e849b0f25d69ef007 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304057727c62dea794c1e849b0f25d69ef007 2023-04-05 04:59:20,060 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040578d374be73464dbdbc1fc502ca613029 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040578d374be73464dbdbc1fc502ca613029 2023-04-05 04:59:20,062 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040580224e8b5c454b23b0d77b7527feafc8 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040580224e8b5c454b23b0d77b7527feafc8 2023-04-05 04:59:20,063 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040586e543e8dc73423bace00750f93bf35c to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040586e543e8dc73423bace00750f93bf35c 2023-04-05 04:59:20,065 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405995e368dec0a42639dd3162662f223ff to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405995e368dec0a42639dd3162662f223ff 2023-04-05 04:59:20,067 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405cc14104005a549aea13f478b2444fe9b to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405cc14104005a549aea13f478b2444fe9b 2023-04-05 04:59:20,068 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405ce09544f32154b7db49a721a5dcea8db to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405ce09544f32154b7db49a721a5dcea8db 2023-04-05 04:59:20,070 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405d58d0a0d78b44972899bbb4478dbc8d8 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405d58d0a0d78b44972899bbb4478dbc8d8 2023-04-05 04:59:20,071 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405da9c17c5ba5243a2abf2c71e6a91b951 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405da9c17c5ba5243a2abf2c71e6a91b951 2023-04-05 04:59:20,073 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405e1e6ccf0033e430e96a95c29131d92a0 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405e1e6ccf0033e430e96a95c29131d92a0 2023-04-05 04:59:20,075 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405e990361585a04399992dcad70329aa20 to hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230405e990361585a04399992dcad70329aa20 2023-04-05 04:59:20,075 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-05 04:59:20,078 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 04:59:20,079 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-05 04:59:20,081 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-05 04:59:20,081 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 04:59:20,081 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-05 04:59:20,082 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680670760081"}]},"ts":"9223372036854775807"} 2023-04-05 04:59:20,083 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-05 04:59:20,083 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => ae76f0bbec3b73d99b1c36f6bc76ae4f, NAME => 'TestAcidGuarantees,,1680670727515.ae76f0bbec3b73d99b1c36f6bc76ae4f.', STARTKEY => '', ENDKEY => ''}] 2023-04-05 04:59:20,083 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-05 04:59:20,083 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680670760083"}]},"ts":"9223372036854775807"} 2023-04-05 04:59:20,084 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-05 04:59:20,085 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-05 04:59:20,086 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 92 msec 2023-04-05 04:59:20,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43797] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-05 04:59:20,098 INFO [Listener at localhost.localdomain/34579] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-05 04:59:20,107 INFO [Listener at localhost.localdomain/34579] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225 (was 225), OpenFileDescriptor=530 (was 543), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=360 (was 382), ProcessCount=166 (was 166), AvailableMemoryMB=7346 (was 7247) - AvailableMemoryMB LEAK? - 2023-04-05 04:59:20,108 INFO [Listener at localhost.localdomain/34579] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-05 04:59:20,108 INFO [Listener at localhost.localdomain/34579] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-05 04:59:20,108 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x70531455 to 127.0.0.1:56229 2023-04-05 04:59:20,109 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:59:20,109 DEBUG [Listener at localhost.localdomain/34579] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-05 04:59:20,109 DEBUG [Listener at localhost.localdomain/34579] util.JVMClusterUtil(257): Found active master hash=1679273787, stopped=false 2023-04-05 04:59:20,109 INFO [Listener at localhost.localdomain/34579] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase20.apache.org,43797,1680670568131 2023-04-05 04:59:20,110 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-05 04:59:20,110 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-05 04:59:20,110 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-05 04:59:20,110 INFO [Listener at localhost.localdomain/34579] procedure2.ProcedureExecutor(629): Stopping 2023-04-05 04:59:20,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-05 04:59:20,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-05 04:59:20,110 DEBUG [Listener at localhost.localdomain/34579] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3eaf8ec6 to 127.0.0.1:56229 2023-04-05 04:59:20,110 DEBUG [Listener at localhost.localdomain/34579] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:59:20,110 INFO [Listener at localhost.localdomain/34579] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase20.apache.org,38091,1680670569206' ***** 2023-04-05 04:59:20,110 INFO [Listener at localhost.localdomain/34579] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-05 04:59:20,111 INFO [RS:0;jenkins-hbase20:38091] regionserver.HeapMemoryManager(220): Stopping 2023-04-05 04:59:20,111 INFO [RS:0;jenkins-hbase20:38091] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-05 04:59:20,111 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-05 04:59:20,111 INFO [RS:0;jenkins-hbase20:38091] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-05 04:59:20,111 INFO [RS:0;jenkins-hbase20:38091] regionserver.HRegionServer(3304): Received CLOSE for 557b26e17f78141e7d0cde2ccb5b79b0 2023-04-05 04:59:20,111 INFO [RS:0;jenkins-hbase20:38091] regionserver.HRegionServer(1145): stopping server jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:20,111 DEBUG [RS:0;jenkins-hbase20:38091] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0c53079d to 127.0.0.1:56229 2023-04-05 04:59:20,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 557b26e17f78141e7d0cde2ccb5b79b0, disabling compactions & flushes 2023-04-05 04:59:20,112 DEBUG [RS:0;jenkins-hbase20:38091] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:59:20,112 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680670571822.557b26e17f78141e7d0cde2ccb5b79b0. 2023-04-05 04:59:20,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680670571822.557b26e17f78141e7d0cde2ccb5b79b0. 2023-04-05 04:59:20,112 INFO [RS:0;jenkins-hbase20:38091] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-05 04:59:20,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680670571822.557b26e17f78141e7d0cde2ccb5b79b0. after waiting 0 ms 2023-04-05 04:59:20,112 INFO [RS:0;jenkins-hbase20:38091] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-05 04:59:20,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680670571822.557b26e17f78141e7d0cde2ccb5b79b0. 2023-04-05 04:59:20,112 INFO [RS:0;jenkins-hbase20:38091] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-05 04:59:20,112 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 557b26e17f78141e7d0cde2ccb5b79b0 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-05 04:59:20,112 INFO [RS:0;jenkins-hbase20:38091] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-05 04:59:20,113 INFO [RS:0;jenkins-hbase20:38091] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-05 04:59:20,113 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-05 04:59:20,113 DEBUG [RS:0;jenkins-hbase20:38091] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, 557b26e17f78141e7d0cde2ccb5b79b0=hbase:namespace,,1680670571822.557b26e17f78141e7d0cde2ccb5b79b0.} 2023-04-05 04:59:20,113 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-05 04:59:20,114 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-05 04:59:20,114 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-05 04:59:20,114 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-05 04:59:20,114 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.94 KB heapSize=36.31 KB 2023-04-05 04:59:20,116 DEBUG [RS:0;jenkins-hbase20:38091] regionserver.HRegionServer(1505): Waiting on 1588230740, 557b26e17f78141e7d0cde2ccb5b79b0 2023-04-05 04:59:20,161 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-05 04:59:20,316 DEBUG [RS:0;jenkins-hbase20:38091] regionserver.HRegionServer(1505): Waiting on 1588230740, 557b26e17f78141e7d0cde2ccb5b79b0 2023-04-05 04:59:20,517 DEBUG [RS:0;jenkins-hbase20:38091] regionserver.HRegionServer(1505): Waiting on 1588230740, 557b26e17f78141e7d0cde2ccb5b79b0 2023-04-05 04:59:20,525 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/hbase/namespace/557b26e17f78141e7d0cde2ccb5b79b0/.tmp/info/adc20a5f8f2b455584658dbbe2bd4654 2023-04-05 04:59:20,526 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.29 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/hbase/meta/1588230740/.tmp/info/85450f27b2c940238b19d8defea3c221 2023-04-05 04:59:20,533 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 85450f27b2c940238b19d8defea3c221 2023-04-05 04:59:20,533 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/hbase/namespace/557b26e17f78141e7d0cde2ccb5b79b0/.tmp/info/adc20a5f8f2b455584658dbbe2bd4654 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/hbase/namespace/557b26e17f78141e7d0cde2ccb5b79b0/info/adc20a5f8f2b455584658dbbe2bd4654 2023-04-05 04:59:20,538 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/hbase/namespace/557b26e17f78141e7d0cde2ccb5b79b0/info/adc20a5f8f2b455584658dbbe2bd4654, entries=2, sequenceid=6, filesize=4.8 K 2023-04-05 04:59:20,539 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 557b26e17f78141e7d0cde2ccb5b79b0 in 427ms, sequenceid=6, compaction requested=false 2023-04-05 04:59:20,545 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/hbase/namespace/557b26e17f78141e7d0cde2ccb5b79b0/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-05 04:59:20,546 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680670571822.557b26e17f78141e7d0cde2ccb5b79b0. 2023-04-05 04:59:20,546 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 557b26e17f78141e7d0cde2ccb5b79b0: 2023-04-05 04:59:20,546 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1680670571822.557b26e17f78141e7d0cde2ccb5b79b0. 2023-04-05 04:59:20,717 DEBUG [RS:0;jenkins-hbase20:38091] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-05 04:59:20,894 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-05 04:59:20,895 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-05 04:59:20,918 DEBUG [RS:0;jenkins-hbase20:38091] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-05 04:59:20,940 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/hbase/meta/1588230740/.tmp/rep_barrier/324949b51b5944b9b72bc1d1aae70daa 2023-04-05 04:59:20,950 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 324949b51b5944b9b72bc1d1aae70daa 2023-04-05 04:59:20,959 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/hbase/meta/1588230740/.tmp/table/2fd63f97db854c838cb483cc4979e5d9 2023-04-05 04:59:20,963 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 2fd63f97db854c838cb483cc4979e5d9 2023-04-05 04:59:20,963 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/hbase/meta/1588230740/.tmp/info/85450f27b2c940238b19d8defea3c221 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/hbase/meta/1588230740/info/85450f27b2c940238b19d8defea3c221 2023-04-05 04:59:20,967 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 85450f27b2c940238b19d8defea3c221 2023-04-05 04:59:20,967 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/hbase/meta/1588230740/info/85450f27b2c940238b19d8defea3c221, entries=22, sequenceid=93, filesize=7.1 K 2023-04-05 04:59:20,969 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/hbase/meta/1588230740/.tmp/rep_barrier/324949b51b5944b9b72bc1d1aae70daa as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/hbase/meta/1588230740/rep_barrier/324949b51b5944b9b72bc1d1aae70daa 2023-04-05 04:59:20,971 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 324949b51b5944b9b72bc1d1aae70daa 2023-04-05 04:59:20,971 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/hbase/meta/1588230740/rep_barrier/324949b51b5944b9b72bc1d1aae70daa, entries=6, sequenceid=93, filesize=5.5 K 2023-04-05 04:59:20,972 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/hbase/meta/1588230740/.tmp/table/2fd63f97db854c838cb483cc4979e5d9 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/hbase/meta/1588230740/table/2fd63f97db854c838cb483cc4979e5d9 2023-04-05 04:59:20,974 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 2fd63f97db854c838cb483cc4979e5d9 2023-04-05 04:59:20,974 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/hbase/meta/1588230740/table/2fd63f97db854c838cb483cc4979e5d9, entries=9, sequenceid=93, filesize=5.5 K 2023-04-05 04:59:20,975 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.94 KB/21446, heapSize ~36.27 KB/37136, currentSize=0 B/0 for 1588230740 in 861ms, sequenceid=93, compaction requested=false 2023-04-05 04:59:20,983 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-05 04:59:20,983 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-05 04:59:20,984 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-05 04:59:20,984 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-05 04:59:20,984 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-05 04:59:21,118 INFO [RS:0;jenkins-hbase20:38091] regionserver.HRegionServer(1171): stopping server jenkins-hbase20.apache.org,38091,1680670569206; all regions closed. 2023-04-05 04:59:21,134 DEBUG [RS:0;jenkins-hbase20:38091] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/oldWALs 2023-04-05 04:59:21,134 INFO [RS:0;jenkins-hbase20:38091] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C38091%2C1680670569206.meta:.meta(num 1680670571588) 2023-04-05 04:59:21,138 DEBUG [RS:0;jenkins-hbase20:38091] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/oldWALs 2023-04-05 04:59:21,138 INFO [RS:0;jenkins-hbase20:38091] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C38091%2C1680670569206:(num 1680670571040) 2023-04-05 04:59:21,138 DEBUG [RS:0;jenkins-hbase20:38091] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:59:21,138 INFO [RS:0;jenkins-hbase20:38091] regionserver.LeaseManager(133): Closed leases 2023-04-05 04:59:21,139 INFO [RS:0;jenkins-hbase20:38091] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-04-05 04:59:21,139 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-05 04:59:21,140 INFO [RS:0;jenkins-hbase20:38091] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:38091 2023-04-05 04:59:21,145 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,38091,1680670569206 2023-04-05 04:59:21,145 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-05 04:59:21,145 ERROR [Listener at localhost.localdomain/34579-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@267b7b23 rejected from java.util.concurrent.ThreadPoolExecutor@2244eb34[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2481] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1374) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-05 04:59:21,146 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-05 04:59:21,146 ERROR [Listener at localhost.localdomain/34579-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2b347314 rejected from java.util.concurrent.ThreadPoolExecutor@2244eb34[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2481] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-05 04:59:21,147 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase20.apache.org,38091,1680670569206] 2023-04-05 04:59:21,148 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase20.apache.org,38091,1680670569206; numProcessing=1 2023-04-05 04:59:21,149 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase20.apache.org,38091,1680670569206 already deleted, retry=false 2023-04-05 04:59:21,149 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase20.apache.org,38091,1680670569206 expired; onlineServers=0 2023-04-05 04:59:21,149 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase20.apache.org,43797,1680670568131' ***** 2023-04-05 04:59:21,149 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-05 04:59:21,149 DEBUG [M:0;jenkins-hbase20:43797] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@252aa500, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-04-05 04:59:21,149 INFO [M:0;jenkins-hbase20:43797] regionserver.HRegionServer(1145): stopping server jenkins-hbase20.apache.org,43797,1680670568131 2023-04-05 04:59:21,150 INFO [M:0;jenkins-hbase20:43797] regionserver.HRegionServer(1171): stopping server jenkins-hbase20.apache.org,43797,1680670568131; all regions closed. 2023-04-05 04:59:21,150 DEBUG [M:0;jenkins-hbase20:43797] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-05 04:59:21,150 DEBUG [M:0;jenkins-hbase20:43797] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-05 04:59:21,150 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-05 04:59:21,150 DEBUG [M:0;jenkins-hbase20:43797] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-05 04:59:21,150 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1680670570735] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1680670570735,5,FailOnTimeoutGroup] 2023-04-05 04:59:21,150 INFO [M:0;jenkins-hbase20:43797] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-05 04:59:21,150 INFO [M:0;jenkins-hbase20:43797] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-05 04:59:21,150 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-05 04:59:21,150 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1680670570734] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1680670570734,5,FailOnTimeoutGroup] 2023-04-05 04:59:21,150 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-05 04:59:21,150 INFO [M:0;jenkins-hbase20:43797] hbase.ChoreService(369): Chore service for: master/jenkins-hbase20:0 had [] on shutdown 2023-04-05 04:59:21,151 DEBUG [M:0;jenkins-hbase20:43797] master.HMaster(1512): Stopping service threads 2023-04-05 04:59:21,151 INFO [M:0;jenkins-hbase20:43797] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-05 04:59:21,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-05 04:59:21,152 ERROR [M:0;jenkins-hbase20:43797] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[IPC Client (1816157726) connection to localhost.localdomain/127.0.0.1:39291 from jenkins,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-05 04:59:21,152 INFO [M:0;jenkins-hbase20:43797] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-05 04:59:21,153 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-05 04:59:21,153 DEBUG [M:0;jenkins-hbase20:43797] zookeeper.ZKUtil(398): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-05 04:59:21,153 WARN [M:0;jenkins-hbase20:43797] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-05 04:59:21,153 INFO [M:0;jenkins-hbase20:43797] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-05 04:59:21,153 INFO [M:0;jenkins-hbase20:43797] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-05 04:59:21,154 DEBUG [M:0;jenkins-hbase20:43797] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-05 04:59:21,154 INFO [M:0;jenkins-hbase20:43797] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-05 04:59:21,154 DEBUG [M:0;jenkins-hbase20:43797] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-05 04:59:21,154 DEBUG [M:0;jenkins-hbase20:43797] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-05 04:59:21,154 DEBUG [M:0;jenkins-hbase20:43797] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-05 04:59:21,154 INFO [M:0;jenkins-hbase20:43797] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=345.14 KB heapSize=393.88 KB 2023-04-05 04:59:21,248 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-05 04:59:21,249 INFO [RS:0;jenkins-hbase20:38091] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase20.apache.org,38091,1680670569206; zookeeper connection closed. 2023-04-05 04:59:21,249 ERROR [Listener at localhost.localdomain/34579-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5373eb90 rejected from java.util.concurrent.ThreadPoolExecutor@2244eb34[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2481] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-05 04:59:21,249 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@60d4d7c2] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@60d4d7c2 2023-04-05 04:59:21,249 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): regionserver:38091-0x1007f7bc4020001, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-05 04:59:21,249 ERROR [Listener at localhost.localdomain/34579-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@f4df17d rejected from java.util.concurrent.ThreadPoolExecutor@2244eb34[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2481] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-05 04:59:21,250 INFO [Listener at localhost.localdomain/34579] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-05 04:59:21,572 INFO [M:0;jenkins-hbase20:43797] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=345.14 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/2c5e068249d048bb8d5f18f651873400 2023-04-05 04:59:21,583 INFO [M:0;jenkins-hbase20:43797] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 2c5e068249d048bb8d5f18f651873400 2023-04-05 04:59:21,585 DEBUG [M:0;jenkins-hbase20:43797] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/2c5e068249d048bb8d5f18f651873400 as hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/2c5e068249d048bb8d5f18f651873400 2023-04-05 04:59:21,592 INFO [M:0;jenkins-hbase20:43797] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 2c5e068249d048bb8d5f18f651873400 2023-04-05 04:59:21,592 INFO [M:0;jenkins-hbase20:43797] regionserver.HStore(1080): Added hdfs://localhost.localdomain:39291/user/jenkins/test-data/d8c6d0b3-cf9c-528b-872e-838ed6e2b6e7/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/2c5e068249d048bb8d5f18f651873400, entries=65, sequenceid=578, filesize=24.7 K 2023-04-05 04:59:21,593 INFO [M:0;jenkins-hbase20:43797] regionserver.HRegion(2948): Finished flush of dataSize ~345.14 KB/353420, heapSize ~393.86 KB/403312, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 439ms, sequenceid=578, compaction requested=false 2023-04-05 04:59:21,595 INFO [M:0;jenkins-hbase20:43797] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-05 04:59:21,595 DEBUG [M:0;jenkins-hbase20:43797] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-05 04:59:21,598 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-05 04:59:21,598 INFO [M:0;jenkins-hbase20:43797] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-05 04:59:21,599 INFO [M:0;jenkins-hbase20:43797] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:43797 2023-04-05 04:59:21,600 DEBUG [M:0;jenkins-hbase20:43797] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase20.apache.org,43797,1680670568131 already deleted, retry=false 2023-04-05 04:59:21,701 INFO [M:0;jenkins-hbase20:43797] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase20.apache.org,43797,1680670568131; zookeeper connection closed. 2023-04-05 04:59:21,701 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-05 04:59:21,701 ERROR [Listener at localhost.localdomain/34579-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1ed62238 rejected from java.util.concurrent.ThreadPoolExecutor@7ed4b9ec[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4154] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-05 04:59:21,702 DEBUG [Listener at localhost.localdomain/34579-EventThread] zookeeper.ZKWatcher(600): master:43797-0x1007f7bc4020000, quorum=127.0.0.1:56229, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-05 04:59:21,702 ERROR [Listener at localhost.localdomain/34579-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@11f8b17b rejected from java.util.concurrent.ThreadPoolExecutor@7ed4b9ec[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4154] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-05 04:59:21,704 WARN [Listener at localhost.localdomain/34579] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-05 04:59:21,709 INFO [Listener at localhost.localdomain/34579] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-05 04:59:21,817 WARN [BP-1103133815-148.251.75.209-1680670565529 heartbeating to localhost.localdomain/127.0.0.1:39291] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-05 04:59:21,817 WARN [BP-1103133815-148.251.75.209-1680670565529 heartbeating to localhost.localdomain/127.0.0.1:39291] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1103133815-148.251.75.209-1680670565529 (Datanode Uuid 6c6dadc0-bae8-4933-9020-feb125a9e7fa) service to localhost.localdomain/127.0.0.1:39291 2023-04-05 04:59:21,821 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bbb152a5-abc2-1c00-5587-f0792a401dc1/cluster_90a8a3c0-d608-d540-0120-10832eef50d1/dfs/data/data1/current/BP-1103133815-148.251.75.209-1680670565529] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-05 04:59:21,821 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bbb152a5-abc2-1c00-5587-f0792a401dc1/cluster_90a8a3c0-d608-d540-0120-10832eef50d1/dfs/data/data2/current/BP-1103133815-148.251.75.209-1680670565529] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-05 04:59:21,855 INFO [Listener at localhost.localdomain/34579] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:0 2023-04-05 04:59:22,010 INFO [Listener at localhost.localdomain/34579] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-05 04:59:22,153 INFO [Listener at localhost.localdomain/34579] hbase.HBaseTestingUtility(1293): Minicluster is down